2024-06-03T04:28:45.429 INFO:root:teuthology version: 0.0.1.dev303+g861a8dc 2024-06-03T04:28:45.429 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/misc} --verbose --archive /home/teuthworker/archive/teuthology-2024-06-02_21:00:03-rados-squid-distro-default-smithi/7738364 --name teuthology-2024-06-02_21:00:03-rados-squid-distro-default-smithi --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-06-02_21:00:03-rados-squid-distro-default-smithi/7738364/orig.config.yaml 2024-06-03T04:28:45.450 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-03T04:28:45.522 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-06-02_21:00:03-rados-squid-distro-default-smithi/7738364 branch: squid description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/misc} email: ceph-qa@ceph.com first_in_suite: false job_id: '7738364' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-06-02_21:00:03-rados-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: squid ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba workunit: branch: squid sha1: dfcc4b3c2c55532c4c04fa47551c9df1dffdf746 owner: scheduled_teuthology@teuthology priority: 101 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 5132 sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba sleep_before_teardown: 0 subset: 20949/100000 suite: rados suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_dfcc4b3c2c55532c4c04fa47551c9df1dffdf746/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: dfcc4b3c2c55532c4c04fa47551c9df1dffdf746 targets: smithi088.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBONEuUnIoiecMBnz3heJ+Gje6wbFfc4Ilb6oVqZnUiAXUVcxlfqS7JXXaMtAK+pZFbDj6ZgTowwUpjf4Reeykb0= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - misc teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 861a8dcf7aa816a26e13f039336f7ed0a9aec0fa timestamp: 2024-06-02_21:00:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841 2024-06-03T04:28:45.522 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_dfcc4b3c2c55532c4c04fa47551c9df1dffdf746/qa; will attempt to use it 2024-06-03T04:28:45.523 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_dfcc4b3c2c55532c4c04fa47551c9df1dffdf746/qa/tasks 2024-06-03T04:28:45.523 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-06-03T04:28:45.525 INFO:teuthology.task.internal:Checking packages... 2024-06-03T04:28:45.545 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '3d89accfe3201286f000e065a1faea2a88c7a6ba' 2024-06-03T04:28:45.545 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-03T04:28:45.546 INFO:teuthology.packaging:ref: None 2024-06-03T04:28:45.546 INFO:teuthology.packaging:tag: None 2024-06-03T04:28:45.546 INFO:teuthology.packaging:branch: squid 2024-06-03T04:28:45.546 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-03T04:28:45.546 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=squid 2024-06-03T04:28:45.811 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2946-g3d89accf-1jammy 2024-06-03T04:28:45.813 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-06-03T04:28:45.822 INFO:teuthology.task.internal:no buildpackages task found 2024-06-03T04:28:45.822 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-06-03T04:28:45.851 INFO:teuthology.task.internal:Saving configuration 2024-06-03T04:28:45.860 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-06-03T04:28:45.878 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-06-03T04:28:45.903 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-06-02_21:00:03-rados-squid-distro-default-smithi/7738364', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-03 04:22:43.270860', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJIemyt5yGPU2hlspHg0+xQ7dDaE5ZR5UtjlDOza5n6b'} 2024-06-03T04:28:45.903 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-06-03T04:28:45.912 INFO:teuthology.task.internal:roles: ubuntu@smithi088.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-06-03T04:28:45.913 INFO:teuthology.run_tasks:Running task console_log... 2024-06-03T04:28:45.992 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f854d660f70>, signals=[15]) 2024-06-03T04:28:45.992 INFO:teuthology.run_tasks:Running task internal.connect... 2024-06-03T04:28:46.003 INFO:teuthology.task.internal:Opening connections... 2024-06-03T04:28:46.003 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi088.front.sepia.ceph.com 2024-06-03T04:28:46.004 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-03T04:28:46.092 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-06-03T04:28:46.098 DEBUG:teuthology.orchestra.run.smithi088:> uname -m 2024-06-03T04:28:46.203 INFO:teuthology.orchestra.run.smithi088.stdout:x86_64 2024-06-03T04:28:46.204 DEBUG:teuthology.orchestra.run.smithi088:> cat /etc/os-release 2024-06-03T04:28:46.248 INFO:teuthology.orchestra.run.smithi088.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-06-03T04:28:46.248 INFO:teuthology.orchestra.run.smithi088.stdout:NAME="Ubuntu" 2024-06-03T04:28:46.249 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION_ID="22.04" 2024-06-03T04:28:46.249 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-06-03T04:28:46.249 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION_CODENAME=jammy 2024-06-03T04:28:46.249 INFO:teuthology.orchestra.run.smithi088.stdout:ID=ubuntu 2024-06-03T04:28:46.249 INFO:teuthology.orchestra.run.smithi088.stdout:ID_LIKE=debian 2024-06-03T04:28:46.249 INFO:teuthology.orchestra.run.smithi088.stdout:HOME_URL="https://www.ubuntu.com/" 2024-06-03T04:28:46.249 INFO:teuthology.orchestra.run.smithi088.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-06-03T04:28:46.249 INFO:teuthology.orchestra.run.smithi088.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-06-03T04:28:46.249 INFO:teuthology.orchestra.run.smithi088.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-06-03T04:28:46.249 INFO:teuthology.orchestra.run.smithi088.stdout:UBUNTU_CODENAME=jammy 2024-06-03T04:28:46.250 INFO:teuthology.lock.ops:Updating smithi088.front.sepia.ceph.com on lock server 2024-06-03T04:28:46.277 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-06-03T04:28:46.290 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-06-03T04:28:46.326 INFO:teuthology.task.internal:Checking for old test directory... 2024-06-03T04:28:46.326 DEBUG:teuthology.orchestra.run.smithi088:> test '!' -e /home/ubuntu/cephtest 2024-06-03T04:28:46.331 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-06-03T04:28:46.394 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-06-03T04:28:46.394 DEBUG:teuthology.orchestra.run.smithi088:> test -z $(ls -A /var/lib/ceph) 2024-06-03T04:28:46.401 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-06-03T04:28:46.477 INFO:teuthology.run_tasks:Running task kernel... 2024-06-03T04:28:46.498 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-06-03T04:28:46.498 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-06-03T04:28:46.499 DEBUG:teuthology.orchestra.run.smithi088:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-03T04:28:46.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-03T04:28:46.504 DEBUG:teuthology.orchestra.run.smithi088:> uname -r 2024-06-03T04:28:46.553 INFO:teuthology.orchestra.run.smithi088.stdout:5.15.0-56-generic 2024-06-03T04:28:46.554 INFO:teuthology.task.kernel:Running kernel on smithi088: 5.15.0-56-generic 2024-06-03T04:28:46.554 DEBUG:teuthology.orchestra.run.smithi088:> sudo apt-get clean 2024-06-03T04:28:46.684 DEBUG:teuthology.orchestra.run.smithi088:> sudo apt-get update 2024-06-03T04:28:46.963 INFO:teuthology.orchestra.run.smithi088.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-06-03T04:28:46.963 INFO:teuthology.orchestra.run.smithi088.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-06-03T04:28:47.028 INFO:teuthology.orchestra.run.smithi088.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-06-03T04:28:47.420 INFO:teuthology.orchestra.run.smithi088.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [471 kB] 2024-06-03T04:28:47.422 INFO:teuthology.orchestra.run.smithi088.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-06-03T04:28:47.572 INFO:teuthology.orchestra.run.smithi088.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,686 kB] 2024-06-03T04:28:47.718 INFO:teuthology.orchestra.run.smithi088.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,475 kB] 2024-06-03T04:28:47.893 INFO:teuthology.orchestra.run.smithi088.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [633 kB] 2024-06-03T04:28:47.896 INFO:teuthology.orchestra.run.smithi088.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [254 kB] 2024-06-03T04:28:47.909 INFO:teuthology.orchestra.run.smithi088.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-06-03T04:28:47.910 INFO:teuthology.orchestra.run.smithi088.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-06-03T04:28:47.912 INFO:teuthology.orchestra.run.smithi088.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,876 kB] 2024-06-03T04:28:47.933 INFO:teuthology.orchestra.run.smithi088.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [313 kB] 2024-06-03T04:28:47.948 INFO:teuthology.orchestra.run.smithi088.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-06-03T04:28:47.949 INFO:teuthology.orchestra.run.smithi088.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,933 kB] 2024-06-03T04:28:47.999 INFO:teuthology.orchestra.run.smithi088.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [318 kB] 2024-06-03T04:28:48.008 INFO:teuthology.orchestra.run.smithi088.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-06-03T04:28:48.047 INFO:teuthology.orchestra.run.smithi088.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-06-03T04:28:48.048 INFO:teuthology.orchestra.run.smithi088.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [854 kB] 2024-06-03T04:28:48.049 INFO:teuthology.orchestra.run.smithi088.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [328 kB] 2024-06-03T04:28:48.059 INFO:teuthology.orchestra.run.smithi088.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-06-03T04:28:48.067 INFO:teuthology.orchestra.run.smithi088.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [604 kB] 2024-06-03T04:28:48.233 INFO:teuthology.orchestra.run.smithi088.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [703 kB] 2024-06-03T04:28:48.250 INFO:teuthology.orchestra.run.smithi088.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [165 kB] 2024-06-03T04:28:48.683 INFO:teuthology.orchestra.run.smithi088.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-06-03T04:28:48.688 INFO:teuthology.orchestra.run.smithi088.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-06-03T04:28:48.704 INFO:teuthology.orchestra.run.smithi088.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-06-03T04:28:48.704 INFO:teuthology.orchestra.run.smithi088.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-06-03T04:28:48.706 INFO:teuthology.orchestra.run.smithi088.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-06-03T04:28:48.853 INFO:teuthology.orchestra.run.smithi088.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,076 kB] 2024-06-03T04:28:48.971 INFO:teuthology.orchestra.run.smithi088.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [247 kB] 2024-06-03T04:28:48.992 INFO:teuthology.orchestra.run.smithi088.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-06-03T04:28:48.994 INFO:teuthology.orchestra.run.smithi088.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-06-03T04:28:48.997 INFO:teuthology.orchestra.run.smithi088.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-06-03T04:28:48.998 INFO:teuthology.orchestra.run.smithi088.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-06-03T04:28:48.999 INFO:teuthology.orchestra.run.smithi088.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-06-03T04:28:49.000 INFO:teuthology.orchestra.run.smithi088.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-06-03T04:28:49.003 INFO:teuthology.orchestra.run.smithi088.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-06-03T04:28:49.007 INFO:teuthology.orchestra.run.smithi088.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-06-03T04:28:49.181 INFO:teuthology.orchestra.run.smithi088.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-06-03T04:28:49.268 INFO:teuthology.orchestra.run.smithi088.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-06-03T04:28:49.354 INFO:teuthology.orchestra.run.smithi088.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB] 2024-06-03T04:28:49.411 INFO:teuthology.orchestra.run.smithi088.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-06-03T04:28:49.465 INFO:teuthology.orchestra.run.smithi088.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-06-03T04:28:53.670 INFO:teuthology.orchestra.run.smithi088.stdout:Fetched 13.8 MB in 3s (5,069 kB/s) 2024-06-03T04:28:54.997 INFO:teuthology.orchestra.run.smithi088.stdout:Reading package lists... 2024-06-03T04:28:55.014 DEBUG:teuthology.orchestra.run.smithi088:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-06-03T04:28:55.076 INFO:teuthology.orchestra.run.smithi088.stdout:Reading package lists... 2024-06-03T04:28:55.277 INFO:teuthology.orchestra.run.smithi088.stdout:Building dependency tree... 2024-06-03T04:28:55.278 INFO:teuthology.orchestra.run.smithi088.stdout:Reading state information... 2024-06-03T04:28:55.454 INFO:teuthology.orchestra.run.smithi088.stdout:The following packages were automatically installed and are no longer required: 2024-06-03T04:28:55.454 INFO:teuthology.orchestra.run.smithi088.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-06-03T04:28:55.454 INFO:teuthology.orchestra.run.smithi088.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-06-03T04:28:55.455 INFO:teuthology.orchestra.run.smithi088.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-06-03T04:28:55.455 INFO:teuthology.orchestra.run.smithi088.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-06-03T04:28:55.455 INFO:teuthology.orchestra.run.smithi088.stdout: python2.7-minimal 2024-06-03T04:28:55.455 INFO:teuthology.orchestra.run.smithi088.stdout:Use 'sudo apt autoremove' to remove them. 2024-06-03T04:28:55.456 INFO:teuthology.orchestra.run.smithi088.stdout:The following additional packages will be installed: 2024-06-03T04:28:55.456 INFO:teuthology.orchestra.run.smithi088.stdout: linux-generic linux-headers-5.15.0-107 linux-headers-5.15.0-107-generic 2024-06-03T04:28:55.456 INFO:teuthology.orchestra.run.smithi088.stdout: linux-headers-generic linux-image-5.15.0-107-generic 2024-06-03T04:28:55.457 INFO:teuthology.orchestra.run.smithi088.stdout: linux-modules-5.15.0-107-generic linux-modules-extra-5.15.0-107-generic 2024-06-03T04:28:55.458 INFO:teuthology.orchestra.run.smithi088.stdout:Suggested packages: 2024-06-03T04:28:55.458 INFO:teuthology.orchestra.run.smithi088.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-06-03T04:28:55.458 INFO:teuthology.orchestra.run.smithi088.stdout:Recommended packages: 2024-06-03T04:28:55.458 INFO:teuthology.orchestra.run.smithi088.stdout: thermald 2024-06-03T04:28:55.501 INFO:teuthology.orchestra.run.smithi088.stdout:The following NEW packages will be installed: 2024-06-03T04:28:55.501 INFO:teuthology.orchestra.run.smithi088.stdout: linux-headers-5.15.0-107 linux-headers-5.15.0-107-generic 2024-06-03T04:28:55.502 INFO:teuthology.orchestra.run.smithi088.stdout: linux-image-5.15.0-107-generic linux-modules-5.15.0-107-generic 2024-06-03T04:28:55.502 INFO:teuthology.orchestra.run.smithi088.stdout: linux-modules-extra-5.15.0-107-generic 2024-06-03T04:28:55.503 INFO:teuthology.orchestra.run.smithi088.stdout:The following packages will be upgraded: 2024-06-03T04:28:55.503 INFO:teuthology.orchestra.run.smithi088.stdout: linux-generic linux-headers-generic linux-image-generic 2024-06-03T04:28:55.561 INFO:teuthology.orchestra.run.smithi088.stdout:3 upgraded, 5 newly installed, 0 to remove and 320 not upgraded. 2024-06-03T04:28:55.561 INFO:teuthology.orchestra.run.smithi088.stdout:Need to get 113 MB of archives. 2024-06-03T04:28:55.561 INFO:teuthology.orchestra.run.smithi088.stdout:After this operation, 583 MB of additional disk space will be used. 2024-06-03T04:28:55.561 INFO:teuthology.orchestra.run.smithi088.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-107-generic amd64 5.15.0-107.117 [22.7 MB] 2024-06-03T04:28:56.040 INFO:teuthology.orchestra.run.smithi088.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-107-generic amd64 5.15.0-107.117 [11.5 MB] 2024-06-03T04:28:56.155 INFO:teuthology.orchestra.run.smithi088.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-107-generic amd64 5.15.0-107.117 [63.9 MB] 2024-06-03T04:28:57.011 INFO:teuthology.orchestra.run.smithi088.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.107.107 [1,694 B] 2024-06-03T04:28:57.011 INFO:teuthology.orchestra.run.smithi088.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.107.107 [2,496 B] 2024-06-03T04:28:57.011 INFO:teuthology.orchestra.run.smithi088.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-107 all 5.15.0-107.117 [12.3 MB] 2024-06-03T04:28:57.132 INFO:teuthology.orchestra.run.smithi088.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-107-generic amd64 5.15.0-107.117 [2,887 kB] 2024-06-03T04:28:57.154 INFO:teuthology.orchestra.run.smithi088.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.107.107 [2,354 B] 2024-06-03T04:28:57.538 INFO:teuthology.orchestra.run.smithi088.stdout:Fetched 113 MB in 2s (69.0 MB/s) 2024-06-03T04:28:57.701 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package linux-modules-5.15.0-107-generic. 2024-06-03T04:28:59.748 INFO:teuthology.orchestra.run.smithi088.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-06-03T04:28:59.752 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../0-linux-modules-5.15.0-107-generic_5.15.0-107.117_amd64.deb ... 2024-06-03T04:29:00.050 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking linux-modules-5.15.0-107-generic (5.15.0-107.117) ... 2024-06-03T04:29:02.076 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package linux-image-5.15.0-107-generic. 2024-06-03T04:29:02.097 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../1-linux-image-5.15.0-107-generic_5.15.0-107.117_amd64.deb ... 2024-06-03T04:29:02.185 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking linux-image-5.15.0-107-generic (5.15.0-107.117) ... 2024-06-03T04:29:02.637 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-107-generic. 2024-06-03T04:29:02.658 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-107-generic_5.15.0-107.117_amd64.deb ... 2024-06-03T04:29:02.687 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking linux-modules-extra-5.15.0-107-generic (5.15.0-107.117) ... 2024-06-03T04:29:09.546 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../3-linux-generic_5.15.0.107.107_amd64.deb ... 2024-06-03T04:29:09.687 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking linux-generic (5.15.0.107.107) over (5.15.0.56.54) ... 2024-06-03T04:29:10.003 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.107.107_amd64.deb ... 2024-06-03T04:29:10.107 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking linux-image-generic (5.15.0.107.107) over (5.15.0.56.54) ... 2024-06-03T04:29:10.369 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package linux-headers-5.15.0-107. 2024-06-03T04:29:10.391 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../5-linux-headers-5.15.0-107_5.15.0-107.117_all.deb ... 2024-06-03T04:29:10.477 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking linux-headers-5.15.0-107 (5.15.0-107.117) ... 2024-06-03T04:29:15.808 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package linux-headers-5.15.0-107-generic. 2024-06-03T04:29:15.846 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../6-linux-headers-5.15.0-107-generic_5.15.0-107.117_amd64.deb ... 2024-06-03T04:29:15.883 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking linux-headers-5.15.0-107-generic (5.15.0-107.117) ... 2024-06-03T04:29:18.468 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.107.107_amd64.deb ... 2024-06-03T04:29:18.593 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking linux-headers-generic (5.15.0.107.107) over (5.15.0.56.54) ... 2024-06-03T04:29:18.942 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up linux-headers-5.15.0-107 (5.15.0-107.117) ... 2024-06-03T04:29:19.048 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up linux-headers-5.15.0-107-generic (5.15.0-107.117) ... 2024-06-03T04:29:19.157 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up linux-headers-generic (5.15.0.107.107) ... 2024-06-03T04:29:19.262 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up linux-image-5.15.0-107-generic (5.15.0-107.117) ... 2024-06-03T04:29:20.575 INFO:teuthology.orchestra.run.smithi088.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-06-03T04:29:20.575 INFO:teuthology.orchestra.run.smithi088.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-06-03T04:29:20.575 INFO:teuthology.orchestra.run.smithi088.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-107-generic 2024-06-03T04:29:20.575 INFO:teuthology.orchestra.run.smithi088.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-107-generic 2024-06-03T04:29:20.747 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up linux-modules-extra-5.15.0-107-generic (5.15.0-107.117) ... 2024-06-03T04:29:22.237 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up linux-modules-5.15.0-107-generic (5.15.0-107.117) ... 2024-06-03T04:29:24.341 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up linux-image-generic (5.15.0.107.107) ... 2024-06-03T04:29:24.466 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up linux-generic (5.15.0.107.107) ... 2024-06-03T04:29:24.593 INFO:teuthology.orchestra.run.smithi088.stdout:Processing triggers for linux-image-5.15.0-107-generic (5.15.0-107.117) ... 2024-06-03T04:29:24.663 INFO:teuthology.orchestra.run.smithi088.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-06-03T04:29:24.664 INFO:teuthology.orchestra.run.smithi088.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-107-generic 2024-06-03T04:29:39.247 INFO:teuthology.orchestra.run.smithi088.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-06-03T04:29:39.247 INFO:teuthology.orchestra.run.smithi088.stdout:Sourcing file `/etc/default/grub' 2024-06-03T04:29:39.266 INFO:teuthology.orchestra.run.smithi088.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-06-03T04:29:39.292 INFO:teuthology.orchestra.run.smithi088.stdout:Generating grub configuration file ... 2024-06-03T04:29:39.617 INFO:teuthology.orchestra.run.smithi088.stdout:Found linux image: /boot/vmlinuz-5.15.0-107-generic 2024-06-03T04:29:39.635 INFO:teuthology.orchestra.run.smithi088.stdout:Found initrd image: /boot/initrd.img-5.15.0-107-generic 2024-06-03T04:29:40.057 INFO:teuthology.orchestra.run.smithi088.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-06-03T04:29:40.063 INFO:teuthology.orchestra.run.smithi088.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-06-03T04:29:40.128 INFO:teuthology.orchestra.run.smithi088.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-06-03T04:29:40.134 INFO:teuthology.orchestra.run.smithi088.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-06-03T04:29:40.285 INFO:teuthology.orchestra.run.smithi088.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-06-03T04:29:40.285 INFO:teuthology.orchestra.run.smithi088.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-06-03T04:29:40.285 INFO:teuthology.orchestra.run.smithi088.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-06-03T04:29:40.330 INFO:teuthology.orchestra.run.smithi088.stdout:done 2024-06-03T04:29:40.787 DEBUG:teuthology.orchestra.run.smithi088:> dpkg -s linux-image-generic 2024-06-03T04:29:40.855 INFO:teuthology.orchestra.run.smithi088.stdout:Package: linux-image-generic 2024-06-03T04:29:40.855 INFO:teuthology.orchestra.run.smithi088.stdout:Status: install ok installed 2024-06-03T04:29:40.855 INFO:teuthology.orchestra.run.smithi088.stdout:Priority: optional 2024-06-03T04:29:40.855 INFO:teuthology.orchestra.run.smithi088.stdout:Section: kernel 2024-06-03T04:29:40.855 INFO:teuthology.orchestra.run.smithi088.stdout:Installed-Size: 21 2024-06-03T04:29:40.855 INFO:teuthology.orchestra.run.smithi088.stdout:Maintainer: Ubuntu Kernel Team 2024-06-03T04:29:40.855 INFO:teuthology.orchestra.run.smithi088.stdout:Architecture: amd64 2024-06-03T04:29:40.856 INFO:teuthology.orchestra.run.smithi088.stdout:Source: linux-meta 2024-06-03T04:29:40.856 INFO:teuthology.orchestra.run.smithi088.stdout:Version: 5.15.0.107.107 2024-06-03T04:29:40.856 INFO:teuthology.orchestra.run.smithi088.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-107), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-06-03T04:29:40.856 INFO:teuthology.orchestra.run.smithi088.stdout:Depends: linux-image-5.15.0-107-generic, linux-modules-extra-5.15.0-107-generic, linux-firmware, intel-microcode, amd64-microcode 2024-06-03T04:29:40.856 INFO:teuthology.orchestra.run.smithi088.stdout:Recommends: thermald 2024-06-03T04:29:40.856 INFO:teuthology.orchestra.run.smithi088.stdout:Description: Generic Linux kernel image 2024-06-03T04:29:40.856 INFO:teuthology.orchestra.run.smithi088.stdout: This package will always depend on the latest generic kernel image 2024-06-03T04:29:40.856 INFO:teuthology.orchestra.run.smithi088.stdout: available. 2024-06-03T04:29:40.856 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-107-generic 2024-06-03T04:29:40.856 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-03T04:29:40.857 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-03T04:29:40.857 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-06-03T04:29:40.857 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi088.front.sepia.ceph.com, path=None, version=distro) 2024-06-03T04:29:40.857 DEBUG:teuthology.orchestra.run.smithi088:> sudo apt-get clean 2024-06-03T04:29:41.020 DEBUG:teuthology.orchestra.run.smithi088:> sudo apt-get update 2024-06-03T04:29:41.140 INFO:teuthology.orchestra.run.smithi088.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-06-03T04:29:41.140 INFO:teuthology.orchestra.run.smithi088.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-06-03T04:29:41.147 INFO:teuthology.orchestra.run.smithi088.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-06-03T04:29:41.166 INFO:teuthology.orchestra.run.smithi088.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-06-03T04:29:42.559 INFO:teuthology.orchestra.run.smithi088.stdout:Reading package lists... 2024-06-03T04:29:42.575 DEBUG:teuthology.orchestra.run.smithi088:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-06-03T04:29:42.628 INFO:teuthology.orchestra.run.smithi088.stdout:Reading package lists... 2024-06-03T04:29:42.807 INFO:teuthology.orchestra.run.smithi088.stdout:Building dependency tree... 2024-06-03T04:29:42.808 INFO:teuthology.orchestra.run.smithi088.stdout:Reading state information... 2024-06-03T04:29:42.966 INFO:teuthology.orchestra.run.smithi088.stdout:linux-image-generic is already the newest version (5.15.0.107.107). 2024-06-03T04:29:42.966 INFO:teuthology.orchestra.run.smithi088.stdout:The following packages were automatically installed and are no longer required: 2024-06-03T04:29:42.966 INFO:teuthology.orchestra.run.smithi088.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-06-03T04:29:42.966 INFO:teuthology.orchestra.run.smithi088.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-06-03T04:29:42.966 INFO:teuthology.orchestra.run.smithi088.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-06-03T04:29:42.967 INFO:teuthology.orchestra.run.smithi088.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-06-03T04:29:42.967 INFO:teuthology.orchestra.run.smithi088.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-06-03T04:29:42.967 INFO:teuthology.orchestra.run.smithi088.stdout:Use 'sudo apt autoremove' to remove them. 2024-06-03T04:29:43.004 INFO:teuthology.orchestra.run.smithi088.stdout:0 upgraded, 0 newly installed, 0 to remove and 320 not upgraded. 2024-06-03T04:29:43.006 DEBUG:teuthology.orchestra.run.smithi088:> dpkg -s linux-image-generic 2024-06-03T04:29:43.063 INFO:teuthology.orchestra.run.smithi088.stdout:Package: linux-image-generic 2024-06-03T04:29:43.063 INFO:teuthology.orchestra.run.smithi088.stdout:Status: install ok installed 2024-06-03T04:29:43.063 INFO:teuthology.orchestra.run.smithi088.stdout:Priority: optional 2024-06-03T04:29:43.063 INFO:teuthology.orchestra.run.smithi088.stdout:Section: kernel 2024-06-03T04:29:43.063 INFO:teuthology.orchestra.run.smithi088.stdout:Installed-Size: 21 2024-06-03T04:29:43.063 INFO:teuthology.orchestra.run.smithi088.stdout:Maintainer: Ubuntu Kernel Team 2024-06-03T04:29:43.063 INFO:teuthology.orchestra.run.smithi088.stdout:Architecture: amd64 2024-06-03T04:29:43.063 INFO:teuthology.orchestra.run.smithi088.stdout:Source: linux-meta 2024-06-03T04:29:43.063 INFO:teuthology.orchestra.run.smithi088.stdout:Version: 5.15.0.107.107 2024-06-03T04:29:43.064 INFO:teuthology.orchestra.run.smithi088.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-107), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-06-03T04:29:43.064 INFO:teuthology.orchestra.run.smithi088.stdout:Depends: linux-image-5.15.0-107-generic, linux-modules-extra-5.15.0-107-generic, linux-firmware, intel-microcode, amd64-microcode 2024-06-03T04:29:43.064 INFO:teuthology.orchestra.run.smithi088.stdout:Recommends: thermald 2024-06-03T04:29:43.064 INFO:teuthology.orchestra.run.smithi088.stdout:Description: Generic Linux kernel image 2024-06-03T04:29:43.064 INFO:teuthology.orchestra.run.smithi088.stdout: This package will always depend on the latest generic kernel image 2024-06-03T04:29:43.064 INFO:teuthology.orchestra.run.smithi088.stdout: available. 2024-06-03T04:29:43.064 DEBUG:teuthology.orchestra.run.smithi088:> mktemp 2024-06-03T04:29:43.108 INFO:teuthology.orchestra.run.smithi088.stdout:/tmp/tmp.LC1mONmefu 2024-06-03T04:29:43.108 DEBUG:teuthology.orchestra.run.smithi088:> sudo cp /boot/grub/grub.cfg /tmp/tmp.LC1mONmefu 2024-06-03T04:29:43.164 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0666 /tmp/tmp.LC1mONmefu 2024-06-03T04:29:43.277 DEBUG:teuthology.orchestra.remote:smithi088:/tmp/tmp.LC1mONmefu is 10KB 2024-06-03T04:29:43.290 DEBUG:teuthology.orchestra.run.smithi088:> rm -fr /tmp/tmp.LC1mONmefu 2024-06-03T04:29:43.296 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-06-03T04:29:43.348 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-06-03T04:29:43.348 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-06-03T04:29:43.348 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-06-03T04:29:43.417 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-107-generic 2024-06-03T04:29:43.417 DEBUG:teuthology.orchestra.run.smithi088:> sudo update-grub 2024-06-03T04:29:44.630 INFO:teuthology.orchestra.run.smithi088.stderr:Sourcing file `/etc/default/grub' 2024-06-03T04:29:44.631 INFO:teuthology.orchestra.run.smithi088.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-06-03T04:29:44.633 INFO:teuthology.orchestra.run.smithi088.stderr:Generating grub configuration file ... 2024-06-03T04:29:44.890 INFO:teuthology.orchestra.run.smithi088.stderr:Found linux image: /boot/vmlinuz-5.15.0-107-generic 2024-06-03T04:29:44.896 INFO:teuthology.orchestra.run.smithi088.stderr:Found initrd image: /boot/initrd.img-5.15.0-107-generic 2024-06-03T04:29:45.196 INFO:teuthology.orchestra.run.smithi088.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-06-03T04:29:45.201 INFO:teuthology.orchestra.run.smithi088.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-06-03T04:29:45.246 INFO:teuthology.orchestra.run.smithi088.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-06-03T04:29:45.251 INFO:teuthology.orchestra.run.smithi088.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-06-03T04:29:45.380 INFO:teuthology.orchestra.run.smithi088.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-06-03T04:29:45.380 INFO:teuthology.orchestra.run.smithi088.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-06-03T04:29:45.380 INFO:teuthology.orchestra.run.smithi088.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-06-03T04:29:45.393 INFO:teuthology.orchestra.run.smithi088.stderr:done 2024-06-03T04:29:45.397 DEBUG:teuthology.orchestra.run.smithi088:> sudo shutdown -r now 2024-06-03T04:30:15.424 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-06-03T04:30:15.425 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-06-03T04:30:15.426 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-03T04:30:49.887 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-06-03T04:30:58.895 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-06-03T04:30:58.896 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-03T04:31:01.950 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-06-03T04:31:13.962 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-06-03T04:31:13.963 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-03T04:31:45.810 DEBUG:teuthology.orchestra.run.smithi088:> true 2024-06-03T04:31:46.650 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-06-03T04:31:46.651 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-107-generic"... 2024-06-03T04:31:46.651 DEBUG:teuthology.orchestra.run.smithi088:> uname -r 2024-06-03T04:31:46.698 INFO:teuthology.orchestra.run.smithi088.stdout:5.15.0-107-generic 2024-06-03T04:31:46.698 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-107-generic vs 5.15.0-107-generic 2024-06-03T04:31:46.698 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-03T04:31:46.698 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-06-03T04:31:47.699 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-06-03T04:31:47.699 DEBUG:teuthology.orchestra.run.smithi088:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-03T04:31:47.764 INFO:teuthology.orchestra.run.smithi088.stdout:ttyS1 2024-06-03T04:31:47.784 DEBUG:teuthology.parallel:result is None 2024-06-03T04:31:47.784 INFO:teuthology.run_tasks:Running task internal.base... 2024-06-03T04:31:47.792 INFO:teuthology.task.internal:Creating test directory... 2024-06-03T04:31:47.792 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-03T04:31:47.798 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-06-03T04:31:47.835 INFO:teuthology.run_tasks:Running task internal.archive... 2024-06-03T04:31:47.876 INFO:teuthology.task.internal:Creating archive directory... 2024-06-03T04:31:47.876 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-03T04:31:47.914 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-06-03T04:31:47.921 INFO:teuthology.task.internal:Enabling coredump saving... 2024-06-03T04:31:47.921 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-03T04:31:47.978 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:31:47.986 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:31:47.988 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-06-03T04:31:47.995 INFO:teuthology.task.internal:Configuring sudo... 2024-06-03T04:31:47.995 DEBUG:teuthology.orchestra.run.smithi088:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-03T04:31:48.047 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-06-03T04:31:48.056 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-06-03T04:31:48.056 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-03T04:31:48.094 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-03T04:31:48.142 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-03T04:31:48.190 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-06-03T04:31:48.190 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-03T04:31:48.259 DEBUG:teuthology.orchestra.run.smithi088:> sudo service rsyslog restart 2024-06-03T04:31:48.336 INFO:teuthology.run_tasks:Running task internal.timer... 2024-06-03T04:31:48.343 INFO:teuthology.task.internal:Starting timer... 2024-06-03T04:31:48.343 INFO:teuthology.run_tasks:Running task pcp... 2024-06-03T04:31:48.375 INFO:teuthology.run_tasks:Running task selinux... 2024-06-03T04:31:48.411 DEBUG:teuthology.task.selinux:Excluding smithi088: OS 'ubuntu' does not support SELinux 2024-06-03T04:31:48.412 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-06-03T04:31:48.412 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-06-03T04:31:48.412 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-06-03T04:31:48.412 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-06-03T04:31:48.517 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-06-03T04:31:48.520 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-06-03T04:31:48.668 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-06-03T04:31:48.679 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-06-03T04:31:48.680 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi088.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-06-03T04:39:20.617 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi088.front.sepia.ceph.com')] 2024-06-03T04:39:20.618 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-06-03T04:39:20.619 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-03T04:39:20.705 DEBUG:teuthology.orchestra.run.smithi088:> true 2024-06-03T04:39:20.786 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-06-03T04:39:20.786 INFO:teuthology.run_tasks:Running task clock... 2024-06-03T04:39:20.797 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-06-03T04:39:20.797 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-03T04:39:20.798 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-03T04:39:20.855 INFO:teuthology.orchestra.run.smithi088.stdout: 3 Jun 04:39:20 ntpd[17593]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-06-03T04:39:20.855 INFO:teuthology.orchestra.run.smithi088.stdout: 3 Jun 04:39:20 ntpd[17593]: Command line: ntpd -gq 2024-06-03T04:39:20.855 INFO:teuthology.orchestra.run.smithi088.stdout: 3 Jun 04:39:20 ntpd[17593]: ---------------------------------------------------- 2024-06-03T04:39:20.856 INFO:teuthology.orchestra.run.smithi088.stdout: 3 Jun 04:39:20 ntpd[17593]: ntp-4 is maintained by Network Time Foundation, 2024-06-03T04:39:20.856 INFO:teuthology.orchestra.run.smithi088.stdout: 3 Jun 04:39:20 ntpd[17593]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-06-03T04:39:20.856 INFO:teuthology.orchestra.run.smithi088.stdout: 3 Jun 04:39:20 ntpd[17593]: corporation. Support and training for ntp-4 are 2024-06-03T04:39:20.856 INFO:teuthology.orchestra.run.smithi088.stdout: 3 Jun 04:39:20 ntpd[17593]: available at https://www.nwtime.org/support 2024-06-03T04:39:20.856 INFO:teuthology.orchestra.run.smithi088.stdout: 3 Jun 04:39:20 ntpd[17593]: ---------------------------------------------------- 2024-06-03T04:39:20.856 INFO:teuthology.orchestra.run.smithi088.stdout: 3 Jun 04:39:20 ntpd[17593]: proto: precision = 0.061 usec (-24) 2024-06-03T04:39:20.856 INFO:teuthology.orchestra.run.smithi088.stdout: 3 Jun 04:39:20 ntpd[17593]: basedate set to 2022-02-04 2024-06-03T04:39:20.857 INFO:teuthology.orchestra.run.smithi088.stdout: 3 Jun 04:39:20 ntpd[17593]: gps base set to 2022-02-06 (week 2196) 2024-06-03T04:39:20.857 INFO:teuthology.orchestra.run.smithi088.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-06-03T04:39:20.857 INFO:teuthology.orchestra.run.smithi088.stdout: 3 Jun 04:39:20 ntpd[17593]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-06-03T04:39:20.857 INFO:teuthology.orchestra.run.smithi088.stdout: 3 Jun 04:39:20 ntpd[17593]: restrict ::: KOD does nothing without LIMITED. 2024-06-03T04:39:20.858 INFO:teuthology.orchestra.run.smithi088.stderr:restrict ::: KOD does nothing without LIMITED. 2024-06-03T04:39:20.858 INFO:teuthology.orchestra.run.smithi088.stdout: 3 Jun 04:39:20 ntpd[17593]: Listen and drop on 0 v6wildcard [::]:123 2024-06-03T04:39:20.859 INFO:teuthology.orchestra.run.smithi088.stdout: 3 Jun 04:39:20 ntpd[17593]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-06-03T04:39:20.859 INFO:teuthology.orchestra.run.smithi088.stdout: 3 Jun 04:39:20 ntpd[17593]: Listen normally on 2 lo 127.0.0.1:123 2024-06-03T04:39:20.859 INFO:teuthology.orchestra.run.smithi088.stdout: 3 Jun 04:39:20 ntpd[17593]: Listen normally on 3 enp3s0f1 172.21.15.88:123 2024-06-03T04:39:20.859 INFO:teuthology.orchestra.run.smithi088.stdout: 3 Jun 04:39:20 ntpd[17593]: Listen normally on 4 lo [::1]:123 2024-06-03T04:39:20.859 INFO:teuthology.orchestra.run.smithi088.stdout: 3 Jun 04:39:20 ntpd[17593]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d42d%5]:123 2024-06-03T04:39:20.859 INFO:teuthology.orchestra.run.smithi088.stdout: 3 Jun 04:39:20 ntpd[17593]: Listening on routing socket on fd #22 for interface updates 2024-06-03T04:39:21.856 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:21 ntpd[17593]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-03T04:39:21.857 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:21 ntpd[17593]: can't open /var/log/ntpstats/rawstats.20240603: Permission denied 2024-06-03T04:39:21.857 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:21 ntpd[17593]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-03T04:39:21.857 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:21 ntpd[17593]: can't open /var/log/ntpstats/rawstats.20240603: Permission denied 2024-06-03T04:39:21.857 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:21 ntpd[17593]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-03T04:39:21.857 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:21 ntpd[17593]: can't open /var/log/ntpstats/rawstats.20240603: Permission denied 2024-06-03T04:39:21.857 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:21 ntpd[17593]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-03T04:39:21.857 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:21 ntpd[17593]: can't open /var/log/ntpstats/peerstats.20240603: Permission denied 2024-06-03T04:39:21.857 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:21 ntpd[17593]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-03T04:39:21.857 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:21 ntpd[17593]: can't open /var/log/ntpstats/rawstats.20240603: Permission denied 2024-06-03T04:39:21.857 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:21 ntpd[17593]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-03T04:39:21.857 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:21 ntpd[17593]: can't open /var/log/ntpstats/peerstats.20240603: Permission denied 2024-06-03T04:39:23.857 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:23 ntpd[17593]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-03T04:39:23.857 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:23 ntpd[17593]: can't open /var/log/ntpstats/rawstats.20240603: Permission denied 2024-06-03T04:39:23.858 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:23 ntpd[17593]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-03T04:39:23.858 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:23 ntpd[17593]: can't open /var/log/ntpstats/rawstats.20240603: Permission denied 2024-06-03T04:39:23.858 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:23 ntpd[17593]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-03T04:39:23.858 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:23 ntpd[17593]: can't open /var/log/ntpstats/rawstats.20240603: Permission denied 2024-06-03T04:39:23.858 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:23 ntpd[17593]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-03T04:39:23.858 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:23 ntpd[17593]: can't open /var/log/ntpstats/rawstats.20240603: Permission denied 2024-06-03T04:39:23.858 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:23 ntpd[17593]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-03T04:39:23.858 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:23 ntpd[17593]: can't open /var/log/ntpstats/peerstats.20240603: Permission denied 2024-06-03T04:39:23.858 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:23 ntpd[17593]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-03T04:39:23.858 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:23 ntpd[17593]: can't open /var/log/ntpstats/rawstats.20240603: Permission denied 2024-06-03T04:39:23.859 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:23 ntpd[17593]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-03T04:39:23.859 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:23 ntpd[17593]: can't open /var/log/ntpstats/peerstats.20240603: Permission denied 2024-06-03T04:39:23.859 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:23 ntpd[17593]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-03T04:39:23.859 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:23 ntpd[17593]: can't open /var/log/ntpstats/rawstats.20240603: Permission denied 2024-06-03T04:39:23.859 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:23 ntpd[17593]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-03T04:39:23.859 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:23 ntpd[17593]: can't open /var/log/ntpstats/peerstats.20240603: Permission denied 2024-06-03T04:39:24.856 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:24 ntpd[17593]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-03T04:39:24.856 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:24 ntpd[17593]: can't open /var/log/ntpstats/rawstats.20240603: Permission denied 2024-06-03T04:39:25.858 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:25 ntpd[17593]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-03T04:39:25.858 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:25 ntpd[17593]: can't open /var/log/ntpstats/rawstats.20240603: Permission denied 2024-06-03T04:39:25.859 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:25 ntpd[17593]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-03T04:39:25.859 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:25 ntpd[17593]: can't open /var/log/ntpstats/rawstats.20240603: Permission denied 2024-06-03T04:39:25.859 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:25 ntpd[17593]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-03T04:39:25.859 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:25 ntpd[17593]: can't open /var/log/ntpstats/rawstats.20240603: Permission denied 2024-06-03T04:39:25.859 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:25 ntpd[17593]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-03T04:39:25.859 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:25 ntpd[17593]: can't open /var/log/ntpstats/rawstats.20240603: Permission denied 2024-06-03T04:39:25.859 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:25 ntpd[17593]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-03T04:39:25.860 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:25 ntpd[17593]: can't open /var/log/ntpstats/peerstats.20240603: Permission denied 2024-06-03T04:39:25.860 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:25 ntpd[17593]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-03T04:39:25.860 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:25 ntpd[17593]: can't open /var/log/ntpstats/rawstats.20240603: Permission denied 2024-06-03T04:39:25.860 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:25 ntpd[17593]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-03T04:39:25.860 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:25 ntpd[17593]: can't open /var/log/ntpstats/peerstats.20240603: Permission denied 2024-06-03T04:39:25.860 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:25 ntpd[17593]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-03T04:39:25.861 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:25 ntpd[17593]: can't open /var/log/ntpstats/rawstats.20240603: Permission denied 2024-06-03T04:39:25.861 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:25 ntpd[17593]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-03T04:39:25.861 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:25 ntpd[17593]: can't open /var/log/ntpstats/peerstats.20240603: Permission denied 2024-06-03T04:39:27.857 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:27 ntpd[17593]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-03T04:39:27.857 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:27 ntpd[17593]: can't open /var/log/ntpstats/rawstats.20240603: Permission denied 2024-06-03T04:39:27.857 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:27 ntpd[17593]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-03T04:39:27.857 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:27 ntpd[17593]: can't open /var/log/ntpstats/rawstats.20240603: Permission denied 2024-06-03T04:39:27.857 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:27 ntpd[17593]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-03T04:39:27.857 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:27 ntpd[17593]: can't open /var/log/ntpstats/rawstats.20240603: Permission denied 2024-06-03T04:39:27.857 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:27 ntpd[17593]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-03T04:39:27.857 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:27 ntpd[17593]: can't open /var/log/ntpstats/rawstats.20240603: Permission denied 2024-06-03T04:39:27.858 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:27 ntpd[17593]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-03T04:39:27.858 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:27 ntpd[17593]: can't open /var/log/ntpstats/peerstats.20240603: Permission denied 2024-06-03T04:39:27.858 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:27 ntpd[17593]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-03T04:39:27.858 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:27 ntpd[17593]: can't open /var/log/ntpstats/rawstats.20240603: Permission denied 2024-06-03T04:39:27.858 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:27 ntpd[17593]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-03T04:39:27.858 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:27 ntpd[17593]: can't open /var/log/ntpstats/peerstats.20240603: Permission denied 2024-06-03T04:39:27.858 INFO:teuthology.orchestra.run.smithi088.stdout: 3 Jun 04:39:27 ntpd[17593]: ntpd: time slew +0.001832 s 2024-06-03T04:39:27.859 INFO:teuthology.orchestra.run.smithi088.stdout:ntpd: time slew +0.001832s 2024-06-03T04:39:27.859 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:27 ntpd[17593]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-06-03T04:39:27.859 INFO:teuthology.orchestra.run.smithi088.stderr: 3 Jun 04:39:27 ntpd[17593]: can't open /var/log/ntpstats/loopstats.20240603: Permission denied 2024-06-03T04:39:27.909 INFO:teuthology.orchestra.run.smithi088.stdout: remote refid st t when poll reach delay offset jitter 2024-06-03T04:39:27.909 INFO:teuthology.orchestra.run.smithi088.stdout:============================================================================== 2024-06-03T04:39:27.909 INFO:teuthology.orchestra.run.smithi088.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-06-03T04:39:27.909 INFO:teuthology.orchestra.run.smithi088.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-06-03T04:39:27.910 INFO:teuthology.orchestra.run.smithi088.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-06-03T04:39:27.910 INFO:teuthology.orchestra.run.smithi088.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-06-03T04:39:27.911 INFO:teuthology.run_tasks:Running task install... 2024-06-03T04:39:27.919 DEBUG:teuthology.task.install:project ceph 2024-06-03T04:39:27.920 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba'}} 2024-06-03T04:39:27.920 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba'} 2024-06-03T04:39:27.920 INFO:teuthology.task.install:Using flavor: default 2024-06-03T04:39:27.925 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-06-03T04:39:27.925 INFO:teuthology.task.install:extra packages: [] 2024-06-03T04:39:27.926 DEBUG:teuthology.orchestra.run.smithi088:> sudo apt-key list | grep Ceph 2024-06-03T04:39:28.005 INFO:teuthology.orchestra.run.smithi088.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-06-03T04:39:28.031 INFO:teuthology.orchestra.run.smithi088.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-06-03T04:39:28.032 INFO:teuthology.orchestra.run.smithi088.stdout:uid [ unknown] Ceph.com (release key) 2024-06-03T04:39:28.032 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-06-03T04:39:28.033 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-03T04:39:28.175 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/ubuntu/jammy/flavors/default/ 2024-06-03T04:39:28.175 INFO:teuthology.task.install.deb:Package version is 19.0.0-2946-g3d89accf-1jammy 2024-06-03T04:39:28.285 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-06-03T04:39:28.285 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-06-03T04:39:28.300 DEBUG:teuthology.orchestra.run.smithi088:> sudo apt-get update 2024-06-03T04:39:28.472 INFO:teuthology.orchestra.run.smithi088.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-06-03T04:39:28.472 INFO:teuthology.orchestra.run.smithi088.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-06-03T04:39:28.486 INFO:teuthology.orchestra.run.smithi088.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-06-03T04:39:28.507 INFO:teuthology.orchestra.run.smithi088.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-06-03T04:39:28.617 INFO:teuthology.orchestra.run.smithi088.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/ubuntu/jammy/flavors/default jammy InRelease 2024-06-03T04:39:28.653 INFO:teuthology.orchestra.run.smithi088.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-06-03T04:39:28.690 INFO:teuthology.orchestra.run.smithi088.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/ubuntu/jammy/flavors/default jammy Release.gpg 2024-06-03T04:39:28.726 INFO:teuthology.orchestra.run.smithi088.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-06-03T04:39:28.768 INFO:teuthology.orchestra.run.smithi088.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,400 B] 2024-06-03T04:39:28.840 INFO:teuthology.orchestra.run.smithi088.stdout:Fetched 28.9 kB in 0s (79.7 kB/s) 2024-06-03T04:39:29.799 INFO:teuthology.orchestra.run.smithi088.stdout:Reading package lists... 2024-06-03T04:39:29.815 DEBUG:teuthology.orchestra.run.smithi088:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2946-g3d89accf-1jammy cephadm=19.0.0-2946-g3d89accf-1jammy ceph-mds=19.0.0-2946-g3d89accf-1jammy ceph-mgr=19.0.0-2946-g3d89accf-1jammy ceph-common=19.0.0-2946-g3d89accf-1jammy ceph-fuse=19.0.0-2946-g3d89accf-1jammy ceph-test=19.0.0-2946-g3d89accf-1jammy ceph-volume=19.0.0-2946-g3d89accf-1jammy radosgw=19.0.0-2946-g3d89accf-1jammy python3-rados=19.0.0-2946-g3d89accf-1jammy python3-rgw=19.0.0-2946-g3d89accf-1jammy python3-cephfs=19.0.0-2946-g3d89accf-1jammy python3-rbd=19.0.0-2946-g3d89accf-1jammy libcephfs2=19.0.0-2946-g3d89accf-1jammy libcephfs-dev=19.0.0-2946-g3d89accf-1jammy librados2=19.0.0-2946-g3d89accf-1jammy librbd1=19.0.0-2946-g3d89accf-1jammy rbd-fuse=19.0.0-2946-g3d89accf-1jammy 2024-06-03T04:39:29.876 INFO:teuthology.orchestra.run.smithi088.stdout:Reading package lists... 2024-06-03T04:39:30.078 INFO:teuthology.orchestra.run.smithi088.stdout:Building dependency tree... 2024-06-03T04:39:30.079 INFO:teuthology.orchestra.run.smithi088.stdout:Reading state information... 2024-06-03T04:39:30.261 INFO:teuthology.orchestra.run.smithi088.stdout:The following packages were automatically installed and are no longer required: 2024-06-03T04:39:30.262 INFO:teuthology.orchestra.run.smithi088.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-06-03T04:39:30.262 INFO:teuthology.orchestra.run.smithi088.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-06-03T04:39:30.262 INFO:teuthology.orchestra.run.smithi088.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-06-03T04:39:30.262 INFO:teuthology.orchestra.run.smithi088.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-06-03T04:39:30.263 INFO:teuthology.orchestra.run.smithi088.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-06-03T04:39:30.263 INFO:teuthology.orchestra.run.smithi088.stdout:Use 'sudo apt autoremove' to remove them. 2024-06-03T04:39:30.263 INFO:teuthology.orchestra.run.smithi088.stdout:The following additional packages will be installed: 2024-06-03T04:39:30.263 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-06-03T04:39:30.263 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-06-03T04:39:30.263 INFO:teuthology.orchestra.run.smithi088.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-06-03T04:39:30.263 INFO:teuthology.orchestra.run.smithi088.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-06-03T04:39:30.264 INFO:teuthology.orchestra.run.smithi088.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-06-03T04:39:30.264 INFO:teuthology.orchestra.run.smithi088.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-06-03T04:39:30.264 INFO:teuthology.orchestra.run.smithi088.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-06-03T04:39:30.265 INFO:teuthology.orchestra.run.smithi088.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-06-03T04:39:30.265 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-06-03T04:39:30.265 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-06-03T04:39:30.265 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-06-03T04:39:30.265 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-06-03T04:39:30.265 INFO:teuthology.orchestra.run.smithi088.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-06-03T04:39:30.265 INFO:teuthology.orchestra.run.smithi088.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-06-03T04:39:30.265 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-06-03T04:39:30.265 INFO:teuthology.orchestra.run.smithi088.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-06-03T04:39:30.265 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-06-03T04:39:30.265 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-06-03T04:39:30.265 INFO:teuthology.orchestra.run.smithi088.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-06-03T04:39:30.265 INFO:teuthology.orchestra.run.smithi088.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-06-03T04:39:30.266 INFO:teuthology.orchestra.run.smithi088.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-06-03T04:39:30.266 INFO:teuthology.orchestra.run.smithi088.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-06-03T04:39:30.266 INFO:teuthology.orchestra.run.smithi088.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-06-03T04:39:30.266 INFO:teuthology.orchestra.run.smithi088.stdout: xmlstarlet zip 2024-06-03T04:39:30.266 INFO:teuthology.orchestra.run.smithi088.stdout:Suggested packages: 2024-06-03T04:39:30.266 INFO:teuthology.orchestra.run.smithi088.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-06-03T04:39:30.266 INFO:teuthology.orchestra.run.smithi088.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-06-03T04:39:30.266 INFO:teuthology.orchestra.run.smithi088.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-06-03T04:39:30.266 INFO:teuthology.orchestra.run.smithi088.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-06-03T04:39:30.267 INFO:teuthology.orchestra.run.smithi088.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-06-03T04:39:30.267 INFO:teuthology.orchestra.run.smithi088.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-06-03T04:39:30.267 INFO:teuthology.orchestra.run.smithi088.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-06-03T04:39:30.267 INFO:teuthology.orchestra.run.smithi088.stdout:Recommended packages: 2024-06-03T04:39:30.267 INFO:teuthology.orchestra.run.smithi088.stdout: btrfs-tools 2024-06-03T04:39:30.367 INFO:teuthology.orchestra.run.smithi088.stdout:The following NEW packages will be installed: 2024-06-03T04:39:30.367 INFO:teuthology.orchestra.run.smithi088.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-06-03T04:39:30.367 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-06-03T04:39:30.367 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-06-03T04:39:30.367 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-06-03T04:39:30.367 INFO:teuthology.orchestra.run.smithi088.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-06-03T04:39:30.367 INFO:teuthology.orchestra.run.smithi088.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-06-03T04:39:30.368 INFO:teuthology.orchestra.run.smithi088.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-06-03T04:39:30.369 INFO:teuthology.orchestra.run.smithi088.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-06-03T04:39:30.369 INFO:teuthology.orchestra.run.smithi088.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-06-03T04:39:30.369 INFO:teuthology.orchestra.run.smithi088.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-06-03T04:39:30.369 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-06-03T04:39:30.369 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-06-03T04:39:30.369 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-06-03T04:39:30.369 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-06-03T04:39:30.369 INFO:teuthology.orchestra.run.smithi088.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-06-03T04:39:30.369 INFO:teuthology.orchestra.run.smithi088.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-06-03T04:39:30.369 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-06-03T04:39:30.369 INFO:teuthology.orchestra.run.smithi088.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-06-03T04:39:30.369 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-06-03T04:39:30.370 INFO:teuthology.orchestra.run.smithi088.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-06-03T04:39:30.370 INFO:teuthology.orchestra.run.smithi088.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-06-03T04:39:30.370 INFO:teuthology.orchestra.run.smithi088.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-06-03T04:39:30.370 INFO:teuthology.orchestra.run.smithi088.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-06-03T04:39:30.370 INFO:teuthology.orchestra.run.smithi088.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-06-03T04:39:30.370 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-06-03T04:39:30.370 INFO:teuthology.orchestra.run.smithi088.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-06-03T04:39:30.433 INFO:teuthology.orchestra.run.smithi088.stdout:0 upgraded, 118 newly installed, 0 to remove and 307 not upgraded. 2024-06-03T04:39:30.433 INFO:teuthology.orchestra.run.smithi088.stdout:Need to get 180 MB of archives. 2024-06-03T04:39:30.433 INFO:teuthology.orchestra.run.smithi088.stdout:After this operation, 822 MB of additional disk space will be used. 2024-06-03T04:39:30.434 INFO:teuthology.orchestra.run.smithi088.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-06-03T04:39:30.509 INFO:teuthology.orchestra.run.smithi088.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2946-g3d89accf-1jammy [3,547 kB] 2024-06-03T04:39:30.545 INFO:teuthology.orchestra.run.smithi088.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-06-03T04:39:30.548 INFO:teuthology.orchestra.run.smithi088.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-06-03T04:39:30.569 INFO:teuthology.orchestra.run.smithi088.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-06-03T04:39:30.639 INFO:teuthology.orchestra.run.smithi088.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-06-03T04:39:30.644 INFO:teuthology.orchestra.run.smithi088.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-06-03T04:39:30.657 INFO:teuthology.orchestra.run.smithi088.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-06-03T04:39:30.660 INFO:teuthology.orchestra.run.smithi088.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-06-03T04:39:30.660 INFO:teuthology.orchestra.run.smithi088.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-06-03T04:39:30.661 INFO:teuthology.orchestra.run.smithi088.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-06-03T04:39:30.669 INFO:teuthology.orchestra.run.smithi088.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-06-03T04:39:30.671 INFO:teuthology.orchestra.run.smithi088.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-06-03T04:39:30.673 INFO:teuthology.orchestra.run.smithi088.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-06-03T04:39:30.674 INFO:teuthology.orchestra.run.smithi088.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-06-03T04:39:30.685 INFO:teuthology.orchestra.run.smithi088.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-06-03T04:39:30.687 INFO:teuthology.orchestra.run.smithi088.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-06-03T04:39:30.690 INFO:teuthology.orchestra.run.smithi088.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-06-03T04:39:30.691 INFO:teuthology.orchestra.run.smithi088.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-06-03T04:39:30.705 INFO:teuthology.orchestra.run.smithi088.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-06-03T04:39:30.706 INFO:teuthology.orchestra.run.smithi088.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-06-03T04:39:30.707 INFO:teuthology.orchestra.run.smithi088.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-06-03T04:39:30.707 INFO:teuthology.orchestra.run.smithi088.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-06-03T04:39:30.707 INFO:teuthology.orchestra.run.smithi088.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-06-03T04:39:30.708 INFO:teuthology.orchestra.run.smithi088.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-06-03T04:39:30.726 INFO:teuthology.orchestra.run.smithi088.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-06-03T04:39:30.726 INFO:teuthology.orchestra.run.smithi088.stdout:Get:27 https://2.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2946-g3d89accf-1jammy [3,218 kB] 2024-06-03T04:39:30.727 INFO:teuthology.orchestra.run.smithi088.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-06-03T04:39:30.727 INFO:teuthology.orchestra.run.smithi088.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-06-03T04:39:30.730 INFO:teuthology.orchestra.run.smithi088.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-06-03T04:39:30.746 INFO:teuthology.orchestra.run.smithi088.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-06-03T04:39:30.747 INFO:teuthology.orchestra.run.smithi088.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-06-03T04:39:30.747 INFO:teuthology.orchestra.run.smithi088.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-06-03T04:39:30.749 INFO:teuthology.orchestra.run.smithi088.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-06-03T04:39:30.749 INFO:teuthology.orchestra.run.smithi088.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-06-03T04:39:30.749 INFO:teuthology.orchestra.run.smithi088.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-06-03T04:39:30.767 INFO:teuthology.orchestra.run.smithi088.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-06-03T04:39:30.768 INFO:teuthology.orchestra.run.smithi088.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-06-03T04:39:30.768 INFO:teuthology.orchestra.run.smithi088.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-06-03T04:39:30.774 INFO:teuthology.orchestra.run.smithi088.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-06-03T04:39:30.787 INFO:teuthology.orchestra.run.smithi088.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-06-03T04:39:30.788 INFO:teuthology.orchestra.run.smithi088.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-06-03T04:39:30.789 INFO:teuthology.orchestra.run.smithi088.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-06-03T04:39:30.790 INFO:teuthology.orchestra.run.smithi088.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-06-03T04:39:30.792 INFO:teuthology.orchestra.run.smithi088.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-06-03T04:39:30.793 INFO:teuthology.orchestra.run.smithi088.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-06-03T04:39:30.808 INFO:teuthology.orchestra.run.smithi088.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-06-03T04:39:30.809 INFO:teuthology.orchestra.run.smithi088.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-06-03T04:39:30.811 INFO:teuthology.orchestra.run.smithi088.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-06-03T04:39:30.812 INFO:teuthology.orchestra.run.smithi088.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-06-03T04:39:30.824 INFO:teuthology.orchestra.run.smithi088.stdout:Get:51 https://2.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2946-g3d89accf-1jammy [711 kB] 2024-06-03T04:39:30.829 INFO:teuthology.orchestra.run.smithi088.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-06-03T04:39:30.831 INFO:teuthology.orchestra.run.smithi088.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-06-03T04:39:30.847 INFO:teuthology.orchestra.run.smithi088.stdout:Get:54 https://2.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2946-g3d89accf-1jammy [356 kB] 2024-06-03T04:39:30.858 INFO:teuthology.orchestra.run.smithi088.stdout:Get:55 https://2.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2946-g3d89accf-1jammy [32.8 kB] 2024-06-03T04:39:30.859 INFO:teuthology.orchestra.run.smithi088.stdout:Get:56 https://2.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2946-g3d89accf-1jammy [180 kB] 2024-06-03T04:39:30.865 INFO:teuthology.orchestra.run.smithi088.stdout:Get:57 https://2.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2946-g3d89accf-1jammy [67.1 kB] 2024-06-03T04:39:30.867 INFO:teuthology.orchestra.run.smithi088.stdout:Get:58 https://2.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2946-g3d89accf-1jammy [334 kB] 2024-06-03T04:39:30.878 INFO:teuthology.orchestra.run.smithi088.stdout:Get:59 https://2.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2946-g3d89accf-1jammy [6,783 kB] 2024-06-03T04:39:30.885 INFO:teuthology.orchestra.run.smithi088.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-06-03T04:39:30.886 INFO:teuthology.orchestra.run.smithi088.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-06-03T04:39:30.888 INFO:teuthology.orchestra.run.smithi088.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-06-03T04:39:30.888 INFO:teuthology.orchestra.run.smithi088.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-06-03T04:39:30.889 INFO:teuthology.orchestra.run.smithi088.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-06-03T04:39:30.908 INFO:teuthology.orchestra.run.smithi088.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-06-03T04:39:30.910 INFO:teuthology.orchestra.run.smithi088.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-06-03T04:39:30.910 INFO:teuthology.orchestra.run.smithi088.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-06-03T04:39:30.926 INFO:teuthology.orchestra.run.smithi088.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-06-03T04:39:30.926 INFO:teuthology.orchestra.run.smithi088.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-06-03T04:39:30.934 INFO:teuthology.orchestra.run.smithi088.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-06-03T04:39:30.935 INFO:teuthology.orchestra.run.smithi088.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-06-03T04:39:30.935 INFO:teuthology.orchestra.run.smithi088.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-06-03T04:39:30.936 INFO:teuthology.orchestra.run.smithi088.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-06-03T04:39:30.936 INFO:teuthology.orchestra.run.smithi088.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-06-03T04:39:30.936 INFO:teuthology.orchestra.run.smithi088.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-06-03T04:39:30.940 INFO:teuthology.orchestra.run.smithi088.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-06-03T04:39:30.954 INFO:teuthology.orchestra.run.smithi088.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-06-03T04:39:30.956 INFO:teuthology.orchestra.run.smithi088.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-06-03T04:39:30.957 INFO:teuthology.orchestra.run.smithi088.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-06-03T04:39:30.975 INFO:teuthology.orchestra.run.smithi088.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-06-03T04:39:30.978 INFO:teuthology.orchestra.run.smithi088.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-06-03T04:39:30.979 INFO:teuthology.orchestra.run.smithi088.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-06-03T04:39:30.979 INFO:teuthology.orchestra.run.smithi088.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-06-03T04:39:30.984 INFO:teuthology.orchestra.run.smithi088.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-06-03T04:39:30.984 INFO:teuthology.orchestra.run.smithi088.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-06-03T04:39:30.987 INFO:teuthology.orchestra.run.smithi088.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-06-03T04:39:30.996 INFO:teuthology.orchestra.run.smithi088.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-06-03T04:39:30.996 INFO:teuthology.orchestra.run.smithi088.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-06-03T04:39:30.997 INFO:teuthology.orchestra.run.smithi088.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-06-03T04:39:31.016 INFO:teuthology.orchestra.run.smithi088.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-06-03T04:39:31.017 INFO:teuthology.orchestra.run.smithi088.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-06-03T04:39:31.024 INFO:teuthology.orchestra.run.smithi088.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-06-03T04:39:31.025 INFO:teuthology.orchestra.run.smithi088.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-06-03T04:39:31.025 INFO:teuthology.orchestra.run.smithi088.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-06-03T04:39:31.027 INFO:teuthology.orchestra.run.smithi088.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-06-03T04:39:31.027 INFO:teuthology.orchestra.run.smithi088.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-06-03T04:39:31.102 INFO:teuthology.orchestra.run.smithi088.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2946-g3d89accf-1jammy [112 kB] 2024-06-03T04:39:31.105 INFO:teuthology.orchestra.run.smithi088.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2946-g3d89accf-1jammy [466 kB] 2024-06-03T04:39:31.119 INFO:teuthology.orchestra.run.smithi088.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2946-g3d89accf-1jammy [25.6 MB] 2024-06-03T04:39:32.067 INFO:teuthology.orchestra.run.smithi088.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2946-g3d89accf-1jammy [5,085 kB] 2024-06-03T04:39:32.143 INFO:teuthology.orchestra.run.smithi088.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2946-g3d89accf-1jammy [244 kB] 2024-06-03T04:39:32.146 INFO:teuthology.orchestra.run.smithi088.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2946-g3d89accf-1jammy [124 kB] 2024-06-03T04:39:32.147 INFO:teuthology.orchestra.run.smithi088.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2946-g3d89accf-1jammy [1,525 kB] 2024-06-03T04:39:32.198 INFO:teuthology.orchestra.run.smithi088.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2946-g3d89accf-1jammy [6,248 kB] 2024-06-03T04:39:32.404 INFO:teuthology.orchestra.run.smithi088.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2946-g3d89accf-1jammy [22.7 MB] 2024-06-03T04:39:33.242 INFO:teuthology.orchestra.run.smithi088.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2946-g3d89accf-1jammy [14.1 kB] 2024-06-03T04:39:33.242 INFO:teuthology.orchestra.run.smithi088.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2946-g3d89accf-1jammy [869 kB] 2024-06-03T04:39:33.259 INFO:teuthology.orchestra.run.smithi088.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2946-g3d89accf-1jammy [2,439 kB] 2024-06-03T04:39:33.300 INFO:teuthology.orchestra.run.smithi088.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2946-g3d89accf-1jammy [791 kB] 2024-06-03T04:39:33.309 INFO:teuthology.orchestra.run.smithi088.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2946-g3d89accf-1jammy [148 kB] 2024-06-03T04:39:33.311 INFO:teuthology.orchestra.run.smithi088.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2946-g3d89accf-1jammy [2,295 kB] 2024-06-03T04:39:33.363 INFO:teuthology.orchestra.run.smithi088.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2946-g3d89accf-1jammy [8,625 kB] 2024-06-03T04:39:33.647 INFO:teuthology.orchestra.run.smithi088.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2946-g3d89accf-1jammy [14.2 kB] 2024-06-03T04:39:33.648 INFO:teuthology.orchestra.run.smithi088.stdout:Get:114 https://2.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2946-g3d89accf-1jammy [50.7 MB] 2024-06-03T04:39:35.564 INFO:teuthology.orchestra.run.smithi088.stdout:Get:115 https://2.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2946-g3d89accf-1jammy [121 kB] 2024-06-03T04:39:35.566 INFO:teuthology.orchestra.run.smithi088.stdout:Get:116 https://2.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2946-g3d89accf-1jammy [38.9 kB] 2024-06-03T04:39:35.568 INFO:teuthology.orchestra.run.smithi088.stdout:Get:117 https://2.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2946-g3d89accf-1jammy [13.4 MB] 2024-06-03T04:39:35.896 INFO:teuthology.orchestra.run.smithi088.stdout:Get:118 https://2.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2946-g3d89accf-1jammy [91.4 kB] 2024-06-03T04:39:36.468 INFO:teuthology.orchestra.run.smithi088.stdout:Fetched 180 MB in 6s (32.5 MB/s) 2024-06-03T04:39:36.610 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-06-03T04:39:36.709 INFO:teuthology.orchestra.run.smithi088.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-06-03T04:39:36.714 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-06-03T04:39:36.800 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-06-03T04:39:37.147 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-06-03T04:39:37.174 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-06-03T04:39:37.213 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-06-03T04:39:37.616 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-06-03T04:39:37.643 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-06-03T04:39:37.691 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-06-03T04:39:38.220 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-06-03T04:39:38.246 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-06-03T04:39:38.295 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-06-03T04:39:38.690 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-06-03T04:39:38.716 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-06-03T04:39:38.756 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-06-03T04:39:39.184 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-06-03T04:39:39.210 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-06-03T04:39:39.259 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-06-03T04:39:39.804 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-06-03T04:39:39.830 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-06-03T04:39:39.870 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-06-03T04:39:40.174 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package librados2. 2024-06-03T04:39:40.200 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../007-librados2_19.0.0-2946-g3d89accf-1jammy_amd64.deb ... 2024-06-03T04:39:40.240 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking librados2 (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:39:40.751 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package librbd1. 2024-06-03T04:39:40.778 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../008-librbd1_19.0.0-2946-g3d89accf-1jammy_amd64.deb ... 2024-06-03T04:39:40.825 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking librbd1 (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:39:41.320 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libcephfs2. 2024-06-03T04:39:41.347 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2946-g3d89accf-1jammy_amd64.deb ... 2024-06-03T04:39:41.395 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libcephfs2 (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:39:41.689 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-rados. 2024-06-03T04:39:41.705 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2946-g3d89accf-1jammy_amd64.deb ... 2024-06-03T04:39:41.747 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-rados (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:39:42.067 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-06-03T04:39:42.093 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2946-g3d89accf-1jammy_all.deb ... 2024-06-03T04:39:42.133 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-ceph-argparse (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:39:42.428 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-cephfs. 2024-06-03T04:39:42.454 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2946-g3d89accf-1jammy_amd64.deb ... 2024-06-03T04:39:42.519 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-cephfs (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:39:42.905 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-ceph-common. 2024-06-03T04:39:42.932 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2946-g3d89accf-1jammy_all.deb ... 2024-06-03T04:39:42.980 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-ceph-common (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:39:43.358 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-wcwidth. 2024-06-03T04:39:43.385 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-06-03T04:39:43.424 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-06-03T04:39:43.752 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-prettytable. 2024-06-03T04:39:43.779 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-06-03T04:39:43.819 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-06-03T04:39:44.105 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-rbd. 2024-06-03T04:39:44.132 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2946-g3d89accf-1jammy_amd64.deb ... 2024-06-03T04:39:44.180 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-rbd (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:39:44.592 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-06-03T04:39:44.618 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-06-03T04:39:44.666 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-06-03T04:39:45.068 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-06-03T04:39:45.086 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-06-03T04:39:45.126 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-06-03T04:39:45.438 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-06-03T04:39:45.453 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-06-03T04:39:45.496 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-06-03T04:39:45.891 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package lua5.1. 2024-06-03T04:39:45.918 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-06-03T04:39:45.966 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-06-03T04:39:46.336 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package lua-any. 2024-06-03T04:39:46.362 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-06-03T04:39:46.410 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking lua-any (27ubuntu1) ... 2024-06-03T04:39:46.688 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package zip. 2024-06-03T04:39:46.715 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-06-03T04:39:46.755 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking zip (3.0-12build2) ... 2024-06-03T04:39:47.075 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package unzip. 2024-06-03T04:39:47.101 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-06-03T04:39:47.141 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-06-03T04:39:47.620 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package luarocks. 2024-06-03T04:39:47.646 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-06-03T04:39:47.695 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-06-03T04:39:48.136 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package librgw2. 2024-06-03T04:39:48.162 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../025-librgw2_19.0.0-2946-g3d89accf-1jammy_amd64.deb ... 2024-06-03T04:39:48.210 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking librgw2 (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:39:48.763 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-rgw. 2024-06-03T04:39:48.778 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2946-g3d89accf-1jammy_amd64.deb ... 2024-06-03T04:39:48.821 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-rgw (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:39:49.466 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package liboath0:amd64. 2024-06-03T04:39:49.492 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-06-03T04:39:49.541 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-06-03T04:39:49.919 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libradosstriper1. 2024-06-03T04:39:49.945 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2946-g3d89accf-1jammy_amd64.deb ... 2024-06-03T04:39:49.993 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libradosstriper1 (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:39:50.455 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-common. 2024-06-03T04:39:50.482 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2946-g3d89accf-1jammy_amd64.deb ... 2024-06-03T04:39:50.521 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-common (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:39:51.960 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-base. 2024-06-03T04:39:51.987 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2946-g3d89accf-1jammy_amd64.deb ... 2024-06-03T04:39:52.036 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-base (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:39:52.648 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-06-03T04:39:52.672 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-06-03T04:39:52.714 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-06-03T04:39:53.017 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-cheroot. 2024-06-03T04:39:53.043 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-06-03T04:39:53.083 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-06-03T04:39:53.411 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-06-03T04:39:53.438 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-06-03T04:39:53.478 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-06-03T04:39:53.781 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-jaraco.text. 2024-06-03T04:39:53.807 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-06-03T04:39:53.847 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-06-03T04:39:54.166 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-06-03T04:39:54.193 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-06-03T04:39:54.241 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-06-03T04:39:54.836 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-tempora. 2024-06-03T04:39:54.862 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-06-03T04:39:54.911 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-06-03T04:39:55.247 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-portend. 2024-06-03T04:39:55.273 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-06-03T04:39:55.314 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-06-03T04:39:55.616 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-06-03T04:39:55.643 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-06-03T04:39:55.683 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-06-03T04:39:56.078 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-cherrypy3. 2024-06-03T04:39:56.104 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-06-03T04:39:56.152 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-06-03T04:39:56.523 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-natsort. 2024-06-03T04:39:56.549 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-06-03T04:39:56.589 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-06-03T04:39:56.900 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-logutils. 2024-06-03T04:39:56.927 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-06-03T04:39:56.967 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-06-03T04:39:57.270 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-markupsafe. 2024-06-03T04:39:57.297 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-06-03T04:39:57.337 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-06-03T04:39:57.589 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-mako. 2024-06-03T04:39:57.604 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-06-03T04:39:57.639 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-06-03T04:39:57.951 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-simplegeneric. 2024-06-03T04:39:57.977 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-06-03T04:39:58.017 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-06-03T04:39:58.303 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-singledispatch. 2024-06-03T04:39:58.330 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-06-03T04:39:58.370 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-06-03T04:39:58.673 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-webob. 2024-06-03T04:39:58.700 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-06-03T04:39:58.739 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-06-03T04:39:59.111 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-waitress. 2024-06-03T04:39:59.138 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-06-03T04:39:59.187 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-06-03T04:39:59.503 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-tempita. 2024-06-03T04:39:59.530 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-06-03T04:39:59.570 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-06-03T04:39:59.847 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-paste. 2024-06-03T04:39:59.874 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-06-03T04:39:59.914 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-06-03T04:40:00.242 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-06-03T04:40:00.269 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-06-03T04:40:00.309 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-06-03T04:40:00.586 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pastedeploy. 2024-06-03T04:40:00.613 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-06-03T04:40:00.653 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-06-03T04:40:00.914 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-webtest. 2024-06-03T04:40:00.930 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-06-03T04:40:00.972 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-06-03T04:40:01.284 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pecan. 2024-06-03T04:40:01.310 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-06-03T04:40:01.350 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-06-03T04:40:01.713 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-certifi. 2024-06-03T04:40:01.740 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-06-03T04:40:01.780 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-06-03T04:40:02.108 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-idna. 2024-06-03T04:40:02.134 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-06-03T04:40:02.174 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-06-03T04:40:02.485 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-urllib3. 2024-06-03T04:40:02.512 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-06-03T04:40:02.552 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-06-03T04:40:02.838 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-requests. 2024-06-03T04:40:02.865 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-06-03T04:40:02.905 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-06-03T04:40:03.166 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-werkzeug. 2024-06-03T04:40:03.192 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-06-03T04:40:03.224 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-06-03T04:40:03.510 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-06-03T04:40:03.537 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2946-g3d89accf-1jammy_all.deb ... 2024-06-03T04:40:03.577 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:40:04.782 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-06-03T04:40:04.809 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2946-g3d89accf-1jammy_amd64.deb ... 2024-06-03T04:40:04.849 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:40:05.127 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-bcrypt. 2024-06-03T04:40:05.153 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-06-03T04:40:05.185 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-06-03T04:40:05.496 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-openssl. 2024-06-03T04:40:05.522 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-06-03T04:40:05.563 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-06-03T04:40:05.924 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-mgr. 2024-06-03T04:40:05.950 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2946-g3d89accf-1jammy_amd64.deb ... 2024-06-03T04:40:06.007 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-mgr (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:40:06.402 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-mon. 2024-06-03T04:40:06.428 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2946-g3d89accf-1jammy_amd64.deb ... 2024-06-03T04:40:06.468 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-mon (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:40:07.097 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-osd. 2024-06-03T04:40:07.123 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2946-g3d89accf-1jammy_amd64.deb ... 2024-06-03T04:40:07.163 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-osd (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:40:08.258 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph. 2024-06-03T04:40:08.285 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../066-ceph_19.0.0-2946-g3d89accf-1jammy_amd64.deb ... 2024-06-03T04:40:08.324 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:40:08.585 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-fuse. 2024-06-03T04:40:08.612 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2946-g3d89accf-1jammy_amd64.deb ... 2024-06-03T04:40:08.652 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-fuse (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:40:09.005 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-mds. 2024-06-03T04:40:09.032 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2946-g3d89accf-1jammy_amd64.deb ... 2024-06-03T04:40:09.071 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-mds (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:40:09.533 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package cephadm. 2024-06-03T04:40:09.559 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../069-cephadm_19.0.0-2946-g3d89accf-1jammy_amd64.deb ... 2024-06-03T04:40:09.624 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking cephadm (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:40:09.944 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-asyncssh. 2024-06-03T04:40:09.970 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-06-03T04:40:10.010 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-06-03T04:40:10.555 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python-babel-localedata. 2024-06-03T04:40:10.570 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-06-03T04:40:10.613 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-06-03T04:40:11.626 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-babel. 2024-06-03T04:40:11.653 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-06-03T04:40:11.692 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-06-03T04:40:12.021 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-jinja2. 2024-06-03T04:40:12.047 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-06-03T04:40:12.087 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-06-03T04:40:12.473 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-06-03T04:40:12.500 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2946-g3d89accf-1jammy_all.deb ... 2024-06-03T04:40:12.540 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:40:12.885 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-repoze.lru. 2024-06-03T04:40:12.911 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-06-03T04:40:12.951 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-06-03T04:40:13.254 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-routes. 2024-06-03T04:40:13.281 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-06-03T04:40:13.320 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-06-03T04:40:13.657 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-06-03T04:40:13.684 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2946-g3d89accf-1jammy_all.deb ... 2024-06-03T04:40:13.723 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:40:14.920 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-06-03T04:40:14.936 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-06-03T04:40:14.978 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-06-03T04:40:15.524 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-joblib. 2024-06-03T04:40:15.551 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-06-03T04:40:15.590 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-06-03T04:40:15.976 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-06-03T04:40:16.004 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-06-03T04:40:16.034 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-06-03T04:40:16.354 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-sklearn. 2024-06-03T04:40:16.381 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-06-03T04:40:16.420 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-06-03T04:40:17.156 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-06-03T04:40:17.184 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2946-g3d89accf-1jammy_all.deb ... 2024-06-03T04:40:17.307 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:40:18.270 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-cachetools. 2024-06-03T04:40:18.296 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-06-03T04:40:18.335 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-06-03T04:40:18.614 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pyasn1. 2024-06-03T04:40:18.641 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-06-03T04:40:18.680 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-06-03T04:40:19.017 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-06-03T04:40:19.043 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-06-03T04:40:19.108 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-06-03T04:40:19.420 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-rsa. 2024-06-03T04:40:19.447 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-06-03T04:40:19.486 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-rsa (4.8-1) ... 2024-06-03T04:40:19.781 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-google-auth. 2024-06-03T04:40:19.808 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-06-03T04:40:19.848 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-06-03T04:40:20.159 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-06-03T04:40:20.187 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-06-03T04:40:20.225 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-06-03T04:40:20.529 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-websocket. 2024-06-03T04:40:20.556 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-06-03T04:40:20.595 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-06-03T04:40:20.915 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-kubernetes. 2024-06-03T04:40:20.942 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-06-03T04:40:20.981 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-06-03T04:40:21.603 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-06-03T04:40:21.620 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2946-g3d89accf-1jammy_all.deb ... 2024-06-03T04:40:21.670 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:40:21.998 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libonig5:amd64. 2024-06-03T04:40:22.025 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-06-03T04:40:22.065 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-06-03T04:40:22.335 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libjq1:amd64. 2024-06-03T04:40:22.362 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-06-03T04:40:22.408 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-06-03T04:40:22.678 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package jq. 2024-06-03T04:40:22.694 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-06-03T04:40:22.736 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-06-03T04:40:22.989 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package socat. 2024-06-03T04:40:23.005 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-06-03T04:40:23.047 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-06-03T04:40:23.369 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package xmlstarlet. 2024-06-03T04:40:23.383 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-06-03T04:40:23.425 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-06-03T04:40:23.772 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-test. 2024-06-03T04:40:23.799 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2946-g3d89accf-1jammy_amd64.deb ... 2024-06-03T04:40:23.837 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-test (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:40:26.375 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-volume. 2024-06-03T04:40:26.402 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2946-g3d89accf-1jammy_all.deb ... 2024-06-03T04:40:26.441 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-volume (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:40:26.745 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libcephfs-dev. 2024-06-03T04:40:26.762 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2946-g3d89accf-1jammy_amd64.deb ... 2024-06-03T04:40:26.803 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libcephfs-dev (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:40:27.131 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package lua-socket:amd64. 2024-06-03T04:40:27.159 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-06-03T04:40:27.198 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-06-03T04:40:27.493 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package lua-sec:amd64. 2024-06-03T04:40:27.510 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-06-03T04:40:27.543 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-06-03T04:40:27.922 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package nvme-cli. 2024-06-03T04:40:27.950 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-06-03T04:40:27.989 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-06-03T04:40:28.402 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package pkg-config. 2024-06-03T04:40:28.430 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-06-03T04:40:28.468 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-06-03T04:40:28.738 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-06-03T04:40:28.754 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-06-03T04:40:28.788 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-06-03T04:40:29.241 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-iniconfig. 2024-06-03T04:40:29.268 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-06-03T04:40:29.307 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-06-03T04:40:29.569 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pastescript. 2024-06-03T04:40:29.596 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-06-03T04:40:29.626 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-06-03T04:40:29.938 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pluggy. 2024-06-03T04:40:29.966 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-06-03T04:40:30.005 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-06-03T04:40:30.316 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-psutil. 2024-06-03T04:40:30.343 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-06-03T04:40:30.382 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-06-03T04:40:30.710 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-py. 2024-06-03T04:40:30.738 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-06-03T04:40:30.776 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-py (1.10.0-1) ... 2024-06-03T04:40:31.097 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pygments. 2024-06-03T04:40:31.125 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-06-03T04:40:31.163 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-06-03T04:40:31.626 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pyinotify. 2024-06-03T04:40:31.653 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-06-03T04:40:31.692 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-06-03T04:40:31.961 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-toml. 2024-06-03T04:40:31.989 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-06-03T04:40:32.028 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-06-03T04:40:32.314 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pytest. 2024-06-03T04:40:32.342 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-06-03T04:40:32.405 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-06-03T04:40:32.717 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-simplejson. 2024-06-03T04:40:32.745 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-06-03T04:40:32.775 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-06-03T04:40:33.028 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-06-03T04:40:33.055 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-06-03T04:40:33.094 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-06-03T04:40:33.915 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package radosgw. 2024-06-03T04:40:33.943 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../116-radosgw_19.0.0-2946-g3d89accf-1jammy_amd64.deb ... 2024-06-03T04:40:33.981 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking radosgw (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:40:34.793 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package rbd-fuse. 2024-06-03T04:40:34.809 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2946-g3d89accf-1jammy_amd64.deb ... 2024-06-03T04:40:34.851 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking rbd-fuse (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:40:35.163 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-06-03T04:40:35.423 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-06-03T04:40:35.549 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-06-03T04:40:35.810 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-06-03T04:40:36.126 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-06-03T04:40:36.554 INFO:teuthology.orchestra.run.smithi088.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-06-03T04:40:36.642 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up cephadm (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:40:37.559 INFO:teuthology.orchestra.run.smithi088.stdout:Adding system user cephadm....done 2024-06-03T04:40:37.716 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-06-03T04:40:37.992 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-06-03T04:40:38.243 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-06-03T04:40:38.361 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-06-03T04:40:38.595 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-06-03T04:40:38.846 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-06-03T04:40:38.972 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-py (1.10.0-1) ... 2024-06-03T04:40:39.299 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-06-03T04:40:39.659 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-06-03T04:40:39.918 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-06-03T04:40:40.186 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-06-03T04:40:40.446 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-06-03T04:40:40.680 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-06-03T04:40:40.965 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-06-03T04:40:41.216 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-ceph-argparse (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:40:41.684 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-06-03T04:40:41.810 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-06-03T04:40:41.927 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-06-03T04:40:42.028 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-06-03T04:40:42.263 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-06-03T04:40:42.647 INFO:teuthology.orchestra.run.smithi088.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-06-03T04:40:42.693 INFO:teuthology.orchestra.run.smithi088.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-06-03T04:40:42.758 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-06-03T04:40:42.883 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-06-03T04:40:43.243 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-06-03T04:40:43.352 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-06-03T04:40:43.620 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-06-03T04:40:43.863 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-06-03T04:40:44.156 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up zip (3.0-12build2) ... 2024-06-03T04:40:44.281 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-06-03T04:40:44.942 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-06-03T04:40:45.201 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-06-03T04:40:45.335 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-06-03T04:40:45.445 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-06-03T04:40:45.737 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-06-03T04:40:46.097 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-06-03T04:40:46.348 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-06-03T04:40:46.716 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-06-03T04:40:47.017 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-06-03T04:40:47.385 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-06-03T04:40:47.620 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-06-03T04:40:47.762 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-ceph-common (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:40:48.038 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-06-03T04:40:48.331 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-06-03T04:40:49.445 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-06-03T04:40:49.672 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-06-03T04:40:49.781 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-toml (0.10.2-1) ... 2024-06-03T04:40:50.031 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-06-03T04:40:50.158 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-06-03T04:40:50.284 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-06-03T04:40:50.577 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-06-03T04:40:50.820 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-06-03T04:40:51.104 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-06-03T04:40:51.230 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-06-03T04:40:51.531 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-06-03T04:40:52.049 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-06-03T04:40:52.308 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-06-03T04:40:52.593 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-06-03T04:40:52.836 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-06-03T04:40:53.095 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-06-03T04:40:53.221 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-06-03T04:40:53.488 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-06-03T04:40:53.606 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-06-03T04:40:53.891 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-06-03T04:40:54.191 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-06-03T04:40:54.443 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-06-03T04:40:54.577 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-06-03T04:40:54.703 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-06-03T04:40:54.784 INFO:teuthology.orchestra.run.smithi088.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-06-03T04:40:55.021 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-06-03T04:40:55.147 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-06-03T04:40:55.540 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-06-03T04:40:55.816 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up lua-any (27ubuntu1) ... 2024-06-03T04:40:55.942 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-portend (3.0.0-1) ... 2024-06-03T04:40:56.185 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-06-03T04:40:56.311 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-06-03T04:40:56.620 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-06-03T04:40:56.946 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-06-03T04:40:57.223 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-06-03T04:40:57.349 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-rsa (4.8-1) ... 2024-06-03T04:40:57.625 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-06-03T04:40:57.901 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-06-03T04:40:58.237 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-06-03T04:40:58.597 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-06-03T04:40:58.882 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-06-03T04:40:59.226 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-06-03T04:40:59.327 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up librados2 (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:40:59.445 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:40:59.571 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-06-03T04:40:59.907 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libcephfs2 (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:41:00.033 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libradosstriper1 (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:41:00.159 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-06-03T04:41:00.435 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up librbd1 (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:41:00.570 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-mgr-modules-core (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:41:00.696 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-fuse (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:41:00.909 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-06-03T04:41:00.909 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-06-03T04:41:01.364 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libcephfs-dev (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:41:01.490 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-rados (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:41:01.616 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up librgw2 (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:41:01.742 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-06-03T04:41:02.895 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-rbd (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:41:03.021 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up rbd-fuse (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:41:03.147 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-rgw (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:41:03.256 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-cephfs (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:41:03.366 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-common (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:41:03.894 INFO:teuthology.orchestra.run.smithi088.stdout:Adding group ceph....done 2024-06-03T04:41:04.203 INFO:teuthology.orchestra.run.smithi088.stdout:Adding system user ceph....done 2024-06-03T04:41:04.364 INFO:teuthology.orchestra.run.smithi088.stdout:Setting system user ceph properties....done 2024-06-03T04:41:04.378 INFO:teuthology.orchestra.run.smithi088.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-06-03T04:41:04.510 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-06-03T04:41:04.817 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-06-03T04:41:05.277 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-test (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:41:05.412 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up radosgw (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:41:05.879 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-06-03T04:41:05.880 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-06-03T04:41:06.330 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-base (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:41:06.673 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-06-03T04:41:07.165 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-mds (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:41:07.738 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-06-03T04:41:07.739 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-06-03T04:41:08.203 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-mgr (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:41:08.422 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-06-03T04:41:08.422 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-06-03T04:41:08.879 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-osd (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:41:09.187 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-06-03T04:41:09.187 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-06-03T04:41:09.664 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:41:09.790 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:41:09.941 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-mon (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:41:10.137 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-06-03T04:41:10.137 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-06-03T04:41:10.600 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-mgr-cephadm (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:41:10.743 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:41:10.878 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-mgr-dashboard (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:41:11.029 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-volume (19.0.0-2946-g3d89accf-1jammy) ... 2024-06-03T04:41:11.373 INFO:teuthology.orchestra.run.smithi088.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-06-03T04:41:11.474 INFO:teuthology.orchestra.run.smithi088.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-06-03T04:41:11.626 INFO:teuthology.orchestra.run.smithi088.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-06-03T04:41:12.719 INFO:teuthology.orchestra.run.smithi088.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-06-03T04:41:13.787 INFO:teuthology.orchestra.run.smithi088.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-06-03T04:41:13.793 DEBUG:teuthology.parallel:result is None 2024-06-03T04:41:13.794 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-03T04:41:13.948 DEBUG:teuthology.orchestra.run.smithi088:> dpkg-query -W -f '${Version}' ceph 2024-06-03T04:41:13.967 INFO:teuthology.orchestra.run.smithi088.stdout:19.0.0-2946-g3d89accf-1jammy 2024-06-03T04:41:13.968 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2946-g3d89accf-1jammy 2024-06-03T04:41:13.968 INFO:teuthology.task.install:The correct ceph version 19.0.0-2946-g3d89accf-1jammy is installed. 2024-06-03T04:41:13.969 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-06-03T04:41:13.970 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-06-03T04:41:13.970 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-03T04:41:14.025 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-06-03T04:41:14.025 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-06-03T04:41:14.025 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/daemon-helper 2024-06-03T04:41:14.079 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-03T04:41:14.131 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-06-03T04:41:14.131 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-06-03T04:41:14.131 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-03T04:41:14.183 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-03T04:41:14.234 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-06-03T04:41:14.235 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-06-03T04:41:14.235 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/stdin-killer 2024-06-03T04:41:14.287 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-03T04:41:14.339 INFO:teuthology.run_tasks:Running task workunit... 2024-06-03T04:41:14.356 INFO:tasks.workunit:Pulling workunits from ref dfcc4b3c2c55532c4c04fa47551c9df1dffdf746 2024-06-03T04:41:14.356 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-06-03T04:41:14.356 INFO:tasks.workunit:timeout=3h 2024-06-03T04:41:14.356 INFO:tasks.workunit:cleanup=True 2024-06-03T04:41:14.357 DEBUG:teuthology.orchestra.run.smithi088:> stat -- /home/ubuntu/cephtest/mnt.0 2024-06-03T04:41:14.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-03T04:41:14.389 INFO:teuthology.orchestra.run.smithi088.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-06-03T04:41:14.389 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-06-03T04:41:14.436 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-06-03T04:41:14.436 DEBUG:teuthology.orchestra.run.smithi088:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-06-03T04:41:14.485 DEBUG:teuthology.orchestra.run.smithi088:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout dfcc4b3c2c55532c4c04fa47551c9df1dffdf746 2024-06-03T04:41:14.534 INFO:tasks.workunit.client.0.smithi088.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-06-03T04:42:26.141 INFO:tasks.workunit.client.0.smithi088.stderr:Note: switching to 'dfcc4b3c2c55532c4c04fa47551c9df1dffdf746'. 2024-06-03T04:42:26.141 INFO:tasks.workunit.client.0.smithi088.stderr: 2024-06-03T04:42:26.141 INFO:tasks.workunit.client.0.smithi088.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-06-03T04:42:26.141 INFO:tasks.workunit.client.0.smithi088.stderr:changes and commit them, and you can discard any commits you make in this 2024-06-03T04:42:26.141 INFO:tasks.workunit.client.0.smithi088.stderr:state without impacting any branches by switching back to a branch. 2024-06-03T04:42:26.141 INFO:tasks.workunit.client.0.smithi088.stderr: 2024-06-03T04:42:26.142 INFO:tasks.workunit.client.0.smithi088.stderr:If you want to create a new branch to retain commits you create, you may 2024-06-03T04:42:26.142 INFO:tasks.workunit.client.0.smithi088.stderr:do so (now or later) by using -c with the switch command. Example: 2024-06-03T04:42:26.142 INFO:tasks.workunit.client.0.smithi088.stderr: 2024-06-03T04:42:26.142 INFO:tasks.workunit.client.0.smithi088.stderr: git switch -c 2024-06-03T04:42:26.142 INFO:tasks.workunit.client.0.smithi088.stderr: 2024-06-03T04:42:26.142 INFO:tasks.workunit.client.0.smithi088.stderr:Or undo this operation with: 2024-06-03T04:42:26.142 INFO:tasks.workunit.client.0.smithi088.stderr: 2024-06-03T04:42:26.142 INFO:tasks.workunit.client.0.smithi088.stderr: git switch - 2024-06-03T04:42:26.142 INFO:tasks.workunit.client.0.smithi088.stderr: 2024-06-03T04:42:26.142 INFO:tasks.workunit.client.0.smithi088.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-06-03T04:42:26.142 INFO:tasks.workunit.client.0.smithi088.stderr: 2024-06-03T04:42:26.142 INFO:tasks.workunit.client.0.smithi088.stderr:HEAD is now at dfcc4b3c2c5 Merge pull request #57814 from zdover23/wip-doc-2024-06-01-backport-57804-to-squid 2024-06-03T04:42:26.145 DEBUG:teuthology.orchestra.run.smithi088:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-06-03T04:42:26.194 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-06-03T04:42:26.194 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-06-03T04:42:26.244 INFO:tasks.workunit:Running workunits matching misc on client.0... 2024-06-03T04:42:26.245 INFO:tasks.workunit:Running workunit misc/mclock-config.sh... 2024-06-03T04:42:26.245 DEBUG:teuthology.orchestra.run.smithi088:workunit test misc/mclock-config.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dfcc4b3c2c55532c4c04fa47551c9df1dffdf746 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh 2024-06-03T04:42:26.309 INFO:tasks.workunit.client.0.smithi088.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-06-03T04:42:26.314 INFO:tasks.workunit.client.0.smithi088.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-06-03T04:42:26.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-03T04:42:26.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-03T04:42:26.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-06-03T04:42:26.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-06-03T04:42:26.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-06-03T04:42:26.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-06-03T04:42:26.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-06-03T04:42:26.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-06-03T04:42:26.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/mclock-config 2024-06-03T04:42:26.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:21: run: local dir=td/mclock-config 2024-06-03T04:42:26.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:22: run: shift 2024-06-03T04:42:26.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:24: run: export CEPH_MON=127.0.0.1:7124 2024-06-03T04:42:26.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:24: run: CEPH_MON=127.0.0.1:7124 2024-06-03T04:42:26.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:25: run: export CEPH_ARGS 2024-06-03T04:42:26.315 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:26: run: uuidgen 2024-06-03T04:42:26.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:26: run: CEPH_ARGS+='--fsid=78136f38-0d6f-4bc0-8fbc-da8924e531b4 --auth-supported=none ' 2024-06-03T04:42:26.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2024-06-03T04:42:26.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:28: run: CEPH_ARGS+='--debug-mclock 20 ' 2024-06-03T04:42:26.316 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: set 2024-06-03T04:42:26.316 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-06-03T04:42:26.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: local 'funcs=TEST_backfill_limit_adjustment_mclock 2024-06-03T04:42:26.318 INFO:tasks.workunit.client.0.smithi088.stderr:TEST_profile_builtin_to_custom 2024-06-03T04:42:26.318 INFO:tasks.workunit.client.0.smithi088.stderr:TEST_profile_custom_to_builtin 2024-06-03T04:42:26.318 INFO:tasks.workunit.client.0.smithi088.stderr:TEST_profile_disallow_builtin_params_modify 2024-06-03T04:42:26.318 INFO:tasks.workunit.client.0.smithi088.stderr:TEST_profile_disallow_builtin_params_override 2024-06-03T04:42:26.321 INFO:tasks.workunit.client.0.smithi088.stderr:TEST_recovery_limit_adjustment_mclock' 2024-06-03T04:42:26.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-06-03T04:42:26.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-06-03T04:42:26.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-06-03T04:42:26.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-06-03T04:42:26.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-03T04:42:26.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T04:42:26.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-03T04:42:26.322 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:42:26.322 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:42:26.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:42:26.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:42:26.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:42:26.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:42:26.323 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:42:26.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:42:26.324 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:42:26.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:42:26.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:42:26.324 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:42:26.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:42:26.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:42:26.324 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:42:26.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:42:26.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:42:26.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T04:42:26.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-03T04:42:26.326 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:42:26.326 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:42:26.326 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:42:26.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20650 2024-06-03T04:42:26.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:42:26.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:42:26.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-06-03T04:42:26.328 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T04:42:26.328 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:42:26.328 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:42:26.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20650 2024-06-03T04:42:26.331 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T04:42:26.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-06-03T04:42:26.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-06-03T04:42:26.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T04:42:26.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-06-03T04:42:26.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_backfill_limit_adjustment_mclock td/mclock-config 2024-06-03T04:42:26.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:236: TEST_backfill_limit_adjustment_mclock: local dir=td/mclock-config 2024-06-03T04:42:26.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:238: TEST_backfill_limit_adjustment_mclock: setup td/mclock-config 2024-06-03T04:42:26.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-06-03T04:42:26.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-06-03T04:42:26.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-03T04:42:26.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T04:42:26.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-03T04:42:26.332 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:42:26.332 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:42:26.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:42:26.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:42:26.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:42:26.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:42:26.333 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:42:26.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:42:26.334 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:42:26.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:42:26.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:42:26.336 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:42:26.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:42:26.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:42:26.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:42:26.337 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:42:26.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:42:26.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T04:42:26.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-03T04:42:26.340 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:42:26.340 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:42:26.340 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:42:26.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20650 2024-06-03T04:42:26.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:42:26.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:42:26.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-06-03T04:42:26.342 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T04:42:26.343 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:42:26.343 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:42:26.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20650 2024-06-03T04:42:26.344 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T04:42:26.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T04:42:26.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T04:42:26.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-06-03T04:42:26.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:239: TEST_backfill_limit_adjustment_mclock: run_mon td/mclock-config a 2024-06-03T04:42:26.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-06-03T04:42:26.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T04:42:26.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T04:42:26.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T04:42:26.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-06-03T04:42:26.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-06-03T04:42:26.865 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T04:42:26.865 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:42:26.865 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:42:26.866 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:42:26.866 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:42:26.866 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:42:26.866 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20650/$cluster-$name.asok' 2024-06-03T04:42:26.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.20650/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-03T04:42:27.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T04:42:27.069 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T04:42:27.069 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:42:27.069 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T04:42:27.069 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T04:42:27.071 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T04:42:27.071 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T04:42:27.071 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T04:42:27.071 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T04:42:27.071 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:42:27.071 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:42:27.071 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:42:27.072 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-mon.a.asok 2024-06-03T04:42:27.072 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:42:27.072 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20650/ceph-mon.a.asok config get fsid 2024-06-03T04:42:27.191 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T04:42:27.192 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:42:27.192 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T04:42:27.192 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T04:42:27.193 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T04:42:27.193 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T04:42:27.193 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T04:42:27.193 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T04:42:27.193 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:42:27.193 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:42:27.193 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:42:27.194 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-mon.a.asok 2024-06-03T04:42:27.194 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:42:27.194 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20650/ceph-mon.a.asok config get mon_host 2024-06-03T04:42:27.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:240: TEST_backfill_limit_adjustment_mclock: run_mgr td/mclock-config x 2024-06-03T04:42:27.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-06-03T04:42:27.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T04:42:27.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T04:42:27.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T04:42:27.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-06-03T04:42:27.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T04:42:28.229 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T04:42:28.229 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:42:28.229 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:42:28.230 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:42:28.230 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:42:28.230 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:42:28.230 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20650/$cluster-$name.asok' 2024-06-03T04:42:28.230 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T04:42:28.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.20650/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T04:42:28.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:242: TEST_backfill_limit_adjustment_mclock: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-06-03T04:42:28.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-06-03T04:42:28.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T04:42:28.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T04:42:28.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T04:42:28.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-06-03T04:42:28.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=78136f38-0d6f-4bc0-8fbc-da8924e531b4 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-06-03T04:42:28.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T04:42:28.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T04:42:28.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T04:42:28.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-06-03T04:42:28.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-06-03T04:42:28.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T04:42:28.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T04:42:28.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-06-03T04:42:28.371 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T04:42:28.371 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:42:28.371 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:42:28.373 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:42:28.373 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:42:28.373 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:42:28.373 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20650/$cluster-$name.asok' 2024-06-03T04:42:28.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20650/$cluster-$name.asok' 2024-06-03T04:42:28.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T04:42:28.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T04:42:28.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T04:42:28.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-06-03T04:42:28.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-06-03T04:42:28.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T04:42:28.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T04:42:28.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T04:42:28.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T04:42:28.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T04:42:28.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-06-03T04:42:28.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-06-03T04:42:28.374 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T04:42:28.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f70f4d98-72e4-443f-aea6-baa71c93684f 2024-06-03T04:42:28.375 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 f70f4d98-72e4-443f-aea6-baa71c93684f 2024-06-03T04:42:28.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f70f4d98-72e4-443f-aea6-baa71c93684f' 2024-06-03T04:42:28.375 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T04:42:28.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC0SV1mpExhFxAA5QsH1r3nyGYtqc6QdjxxPg== 2024-06-03T04:42:28.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC0SV1mpExhFxAA5QsH1r3nyGYtqc6QdjxxPg=="}' 2024-06-03T04:42:28.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f70f4d98-72e4-443f-aea6-baa71c93684f -i td/mclock-config/0/new.json 2024-06-03T04:42:28.811 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T04:42:28.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-06-03T04:42:28.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=78136f38-0d6f-4bc0-8fbc-da8924e531b4 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.20650/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQC0SV1mpExhFxAA5QsH1r3nyGYtqc6QdjxxPg== --osd-uuid f70f4d98-72e4-443f-aea6-baa71c93684f 2024-06-03T04:42:28.879 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:42:28.875+0000 7fc42ef017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:42:28.927 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:42:28.923+0000 7fc42ef017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:42:28.928 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:42:28.927+0000 7fc42ef017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:42:28.929 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:42:28.927+0000 7fc42ef017c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-06-03T04:42:28.929 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:42:28.927+0000 7fc42ef017c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-06-03T04:42:28.929 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:42:28.927+0000 7fc42ef017c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-06-03T04:42:29.077 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:42:29.075+0000 7fc42ef017c0 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-06-03T04:42:35.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-06-03T04:42:35.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T04:42:35.335 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T04:42:35.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T04:42:35.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T04:42:35.564 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T04:42:35.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T04:42:35.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T04:42:35.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=78136f38-0d6f-4bc0-8fbc-da8924e531b4 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.20650/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-06-03T04:42:35.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T04:42:35.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T04:42:35.594 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:42:35.591+0000 7f165cb917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:42:35.601 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:42:35.599+0000 7f165cb917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:42:35.602 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:42:35.599+0000 7f165cb917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:42:35.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T04:42:35.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T04:42:35.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T04:42:35.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T04:42:35.795 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T04:42:35.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T04:42:35.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:42:35.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T04:42:35.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:42:35.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:42:36.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:42:36.163 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:42:36.159+0000 7f165cb917c0 -1 Falling back to public interface 2024-06-03T04:42:37.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:42:37.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:42:37.019 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T04:42:37.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T04:42:37.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:42:37.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:42:37.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:42:38.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:42:38.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:42:38.322 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T04:42:38.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T04:42:38.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:42:38.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:42:38.688 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:42:38.687+0000 7f165cb917c0 -1 osd.0 0 log_to_monitors true 2024-06-03T04:42:38.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:42:39.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:42:39.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:42:39.718 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T04:42:39.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T04:42:39.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:42:39.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:42:40.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:42:41.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:42:41.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:42:41.162 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T04:42:41.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T04:42:41.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:42:41.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:42:41.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:42:42.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:42:42.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:42:42.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T04:42:42.627 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T04:42:42.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:42:42.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:42:43.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:42:44.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:42:44.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:42:44.065 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T04:42:44.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T04:42:44.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:42:44.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:42:44.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:42:45.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:42:45.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:42:45.853 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T04:42:45.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T04:42:45.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:42:45.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:42:46.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:42:47.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:42:47.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:42:47.321 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T04:42:47.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T04:42:47.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:42:47.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:42:47.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:42:48.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:42:48.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:42:48.812 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T04:42:48.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T04:42:48.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:42:48.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:42:49.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:42:50.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:42:50.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:42:50.277 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T04:42:50.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T04:42:50.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:42:50.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:42:50.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:42:51.730 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T04:42:51.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:42:51.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:42:51.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T04:42:51.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:42:51.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:42:52.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:42:53.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:42:53.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:42:53.351 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T04:42:53.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T04:42:53.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:42:53.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:42:53.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:42:54.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:42:54.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:42:54.967 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T04:42:54.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T04:42:54.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:42:54.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:42:56.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:42:57.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:42:57.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:42:57.199 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T04:42:57.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T04:42:57.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:42:57.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:42:58.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:42:59.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:42:59.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:42:59.642 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T04:42:59.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T04:42:59.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:42:59.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:43:01.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:43:02.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:43:02.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:43:02.035 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T04:43:02.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T04:43:02.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:43:02.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:43:02.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:43:03.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:43:03.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:43:03.517 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T04:43:03.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T04:43:03.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:43:03.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:43:04.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:43:05.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:43:05.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:43:05.690 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-06-03T04:43:05.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T04:43:05.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:43:05.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:43:06.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:43:07.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:43:07.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:43:07.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-03T04:43:07.609 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-06-03T04:43:07.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:43:07.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:43:08.066 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1464440997,v1:127.0.0.1:6801/1464440997] [v2:127.0.0.1:6802/1464440997,v1:127.0.0.1:6803/1464440997] exists,up f70f4d98-72e4-443f-aea6-baa71c93684f 2024-06-03T04:43:08.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T04:43:08.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T04:43:08.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T04:43:08.067 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-06-03T04:43:08.068 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: bc 2024-06-03T04:43:08.068 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-06-03T04:43:08.069 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:43:08.069 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:43:08.069 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:43:08.069 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:43:08.069 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:43:08.069 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-osd.0.asok 2024-06-03T04:43:08.069 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-06-03T04:43:08.070 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.20650/ceph-osd.0.asok config get osd_max_backfills 2024-06-03T04:43:08.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: local backfills=1 2024-06-03T04:43:08.186 INFO:tasks.workunit.client.0.smithi088.stdout:osd_max_backfills: 1 2024-06-03T04:43:08.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:246: TEST_backfill_limit_adjustment_mclock: echo 'osd_max_backfills: 1' 2024-06-03T04:43:08.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:251: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_max_backfills 20 2024-06-03T04:43:08.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:252: TEST_backfill_limit_adjustment_mclock: sleep 2 2024-06-03T04:43:10.784 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-06-03T04:43:10.784 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: bc 2024-06-03T04:43:10.784 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-06-03T04:43:10.784 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:43:10.785 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:43:10.785 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:43:10.785 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:43:10.785 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:43:10.785 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-osd.0.asok 2024-06-03T04:43:10.786 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-06-03T04:43:10.786 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.20650/ceph-osd.0.asok config get osd_max_backfills 2024-06-03T04:43:10.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: local max_backfills=1 2024-06-03T04:43:10.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:255: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2024-06-03T04:43:10.911 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2024-06-03T04:43:10.911 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: bc 2024-06-03T04:43:10.911 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-06-03T04:43:10.911 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:43:10.911 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:43:10.912 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:43:10.912 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:43:10.912 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:43:10.912 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-osd.0.asok 2024-06-03T04:43:10.912 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-06-03T04:43:10.913 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.20650/ceph-osd.0.asok dump_recovery_reservations 2024-06-03T04:43:11.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: max_backfills=1 2024-06-03T04:43:11.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:260: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2024-06-03T04:43:11.027 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2024-06-03T04:43:11.027 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: bc 2024-06-03T04:43:11.029 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-06-03T04:43:11.029 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:43:11.029 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:43:11.029 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:43:11.029 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:43:11.029 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:43:11.029 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-osd.0.asok 2024-06-03T04:43:11.029 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-06-03T04:43:11.030 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.20650/ceph-osd.0.asok dump_recovery_reservations 2024-06-03T04:43:11.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: max_backfills=1 2024-06-03T04:43:11.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:263: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2024-06-03T04:43:11.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:267: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_mclock_override_recovery_settings true 2024-06-03T04:43:11.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:268: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_max_backfills 20 2024-06-03T04:43:12.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:269: TEST_backfill_limit_adjustment_mclock: sleep 2 2024-06-03T04:43:14.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-06-03T04:43:14.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: bc 2024-06-03T04:43:14.276 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-06-03T04:43:14.276 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:43:14.276 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:43:14.277 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:43:14.277 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:43:14.277 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:43:14.277 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-osd.0.asok 2024-06-03T04:43:14.277 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-06-03T04:43:14.277 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.20650/ceph-osd.0.asok config get osd_max_backfills 2024-06-03T04:43:14.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-06-03T04:43:14.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:272: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-06-03T04:43:14.395 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2024-06-03T04:43:14.396 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: bc 2024-06-03T04:43:14.397 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-06-03T04:43:14.397 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:43:14.397 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:43:14.397 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:43:14.397 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:43:14.397 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:43:14.397 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-osd.0.asok 2024-06-03T04:43:14.398 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-06-03T04:43:14.398 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.20650/ceph-osd.0.asok dump_recovery_reservations 2024-06-03T04:43:14.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-06-03T04:43:14.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:277: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-06-03T04:43:14.509 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2024-06-03T04:43:14.509 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: bc 2024-06-03T04:43:14.510 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-06-03T04:43:14.510 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:43:14.510 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:43:14.511 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:43:14.511 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:43:14.511 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:43:14.511 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-osd.0.asok 2024-06-03T04:43:14.511 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-06-03T04:43:14.511 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.20650/ceph-osd.0.asok dump_recovery_reservations 2024-06-03T04:43:14.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-06-03T04:43:14.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:280: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-06-03T04:43:14.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:284: TEST_backfill_limit_adjustment_mclock: kill_daemons td/mclock-config TERM osd 2024-06-03T04:43:14.617 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:43:14.617 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:43:14.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:43:14.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:43:14.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:43:14.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:43:14.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:285: TEST_backfill_limit_adjustment_mclock: ceph osd down 0 2024-06-03T04:43:15.361 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already down. 2024-06-03T04:43:15.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:286: TEST_backfill_limit_adjustment_mclock: wait_for_osd down 0 2024-06-03T04:43:15.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2024-06-03T04:43:15.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T04:43:15.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T04:43:15.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T04:43:15.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:43:15.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T04:43:15.380 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T04:43:15.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:43:15.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2024-06-03T04:43:15.811 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 7 last_clean_interval [0,0) [v2:127.0.0.1:6800/1464440997,v1:127.0.0.1:6801/1464440997] [v2:127.0.0.1:6802/1464440997,v1:127.0.0.1:6803/1464440997] exists f70f4d98-72e4-443f-aea6-baa71c93684f 2024-06-03T04:43:15.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T04:43:15.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T04:43:15.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T04:43:15.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:287: TEST_backfill_limit_adjustment_mclock: activate_osd td/mclock-config 0 --osd-op-queue=mclock_scheduler 2024-06-03T04:43:15.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/mclock-config 2024-06-03T04:43:15.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T04:43:15.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T04:43:15.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T04:43:15.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/mclock-config/0 2024-06-03T04:43:15.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=78136f38-0d6f-4bc0-8fbc-da8924e531b4 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-06-03T04:43:15.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T04:43:15.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T04:43:15.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T04:43:15.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-06-03T04:43:15.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-06-03T04:43:15.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T04:43:15.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T04:43:15.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-06-03T04:43:15.816 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T04:43:15.816 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:43:15.816 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:43:15.816 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:43:15.816 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:43:15.816 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:43:15.816 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20650/$cluster-$name.asok' 2024-06-03T04:43:15.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20650/$cluster-$name.asok' 2024-06-03T04:43:15.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T04:43:15.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-06-03T04:43:15.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-06-03T04:43:15.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T04:43:15.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T04:43:15.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T04:43:15.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T04:43:15.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T04:43:15.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=--osd-op-queue=mclock_scheduler 2024-06-03T04:43:15.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/mclock-config/0 2024-06-03T04:43:15.817 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T04:43:15.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T04:43:15.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=78136f38-0d6f-4bc0-8fbc-da8924e531b4 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.20650/$cluster-$name.asok' --debug-osd=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler 2024-06-03T04:43:15.819 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/mclock-config/0/whoami 2024-06-03T04:43:15.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T04:43:15.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T04:43:15.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T04:43:15.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T04:43:15.844 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:43:15.843+0000 7f926bd9c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:43:15.846 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:43:15.843+0000 7f926bd9c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:43:15.847 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:43:15.843+0000 7f926bd9c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:43:16.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T04:43:16.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T04:43:16.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T04:43:16.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T04:43:16.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T04:43:16.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:43:16.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T04:43:16.220 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T04:43:16.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:43:16.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:43:16.382 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:43:16.379+0000 7f926bd9c7c0 -1 Falling back to public interface 2024-06-03T04:43:17.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:43:18.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:43:18.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:43:18.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T04:43:18.008 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T04:43:18.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:43:18.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:43:18.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:43:19.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:43:19.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:43:19.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T04:43:19.420 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T04:43:19.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:43:19.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:43:19.652 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:43:19.651+0000 7f926bd9c7c0 -1 osd.0 6 log_to_monitors true 2024-06-03T04:43:19.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:43:20.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:43:20.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:43:20.854 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T04:43:20.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T04:43:20.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:43:20.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:43:21.275 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6800/2330842391,v1:127.0.0.1:6801/2330842391] [v2:127.0.0.1:6802/2330842391,v1:127.0.0.1:6803/2330842391] exists,up f70f4d98-72e4-443f-aea6-baa71c93684f 2024-06-03T04:43:21.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T04:43:21.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T04:43:21.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T04:43:21.277 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-06-03T04:43:21.277 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: bc 2024-06-03T04:43:21.277 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-06-03T04:43:21.277 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:43:21.277 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:43:21.278 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:43:21.278 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:43:21.278 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:43:21.278 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-osd.0.asok 2024-06-03T04:43:21.279 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-06-03T04:43:21.279 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.20650/ceph-osd.0.asok config get osd_max_backfills 2024-06-03T04:43:21.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-06-03T04:43:21.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:291: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-06-03T04:43:21.387 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: bc 2024-06-03T04:43:21.387 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2024-06-03T04:43:21.387 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-06-03T04:43:21.387 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:43:21.387 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:43:21.387 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:43:21.387 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:43:21.388 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:43:21.388 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-osd.0.asok 2024-06-03T04:43:21.388 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-06-03T04:43:21.388 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.20650/ceph-osd.0.asok dump_recovery_reservations 2024-06-03T04:43:21.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-06-03T04:43:21.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:296: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-06-03T04:43:21.497 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2024-06-03T04:43:21.497 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-06-03T04:43:21.497 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:43:21.497 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:43:21.497 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: bc 2024-06-03T04:43:21.497 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:43:21.498 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:43:21.498 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:43:21.498 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-osd.0.asok 2024-06-03T04:43:21.498 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-06-03T04:43:21.498 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.20650/ceph-osd.0.asok dump_recovery_reservations 2024-06-03T04:43:21.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-06-03T04:43:21.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:299: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-06-03T04:43:21.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:301: TEST_backfill_limit_adjustment_mclock: teardown td/mclock-config 2024-06-03T04:43:21.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-03T04:43:21.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T04:43:21.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-03T04:43:21.616 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:43:21.616 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:43:21.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:43:21.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:43:21.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:43:21.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:43:21.723 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:43:21.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:43:21.724 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:43:21.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:43:21.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:43:21.725 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:43:21.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:43:21.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:43:21.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:43:21.726 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:43:21.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:43:21.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T04:43:21.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-03T04:43:21.734 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:43:21.734 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:43:21.734 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:43:21.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20650 2024-06-03T04:43:21.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:43:21.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:43:21.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-06-03T04:43:21.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-03T04:43:21.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T04:43:21.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-03T04:43:21.736 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:43:21.736 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:43:21.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:43:21.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:43:21.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:43:21.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:43:21.737 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:43:21.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:43:21.738 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:43:21.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:43:21.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:43:21.740 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:43:21.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:43:21.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:43:21.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:43:21.741 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:43:21.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:43:21.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T04:43:21.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-03T04:43:21.743 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:43:21.743 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:43:21.743 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:43:21.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20650 2024-06-03T04:43:21.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:43:21.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:43:21.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-06-03T04:43:21.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-06-03T04:43:21.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-06-03T04:43:21.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-06-03T04:43:21.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-03T04:43:21.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T04:43:21.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-03T04:43:21.746 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:43:21.746 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:43:21.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:43:21.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:43:21.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:43:21.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:43:21.747 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:43:21.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:43:21.748 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:43:21.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:43:21.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:43:21.749 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:43:21.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:43:21.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:43:21.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:43:21.750 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:43:21.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:43:21.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T04:43:21.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-03T04:43:21.753 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:43:21.753 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:43:21.753 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:43:21.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20650 2024-06-03T04:43:21.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:43:21.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:43:21.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-06-03T04:43:21.755 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T04:43:21.755 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:43:21.755 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:43:21.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20650 2024-06-03T04:43:21.757 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T04:43:21.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T04:43:21.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T04:43:21.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-06-03T04:43:21.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_builtin_to_custom td/mclock-config 2024-06-03T04:43:21.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:39: TEST_profile_builtin_to_custom: local dir=td/mclock-config 2024-06-03T04:43:21.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:41: TEST_profile_builtin_to_custom: run_mon td/mclock-config a 2024-06-03T04:43:21.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-06-03T04:43:21.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T04:43:21.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T04:43:21.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T04:43:21.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-06-03T04:43:21.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-06-03T04:43:22.167 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T04:43:22.167 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:43:22.167 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:43:22.168 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:43:22.168 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:43:22.168 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:43:22.168 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20650/$cluster-$name.asok' 2024-06-03T04:43:22.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.20650/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-03T04:43:22.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T04:43:22.396 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T04:43:22.396 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:43:22.396 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T04:43:22.396 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T04:43:22.398 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T04:43:22.398 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T04:43:22.398 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T04:43:22.398 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T04:43:22.398 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:43:22.398 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:43:22.398 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:43:22.398 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-mon.a.asok 2024-06-03T04:43:22.398 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:43:22.398 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20650/ceph-mon.a.asok config get fsid 2024-06-03T04:43:22.509 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T04:43:22.509 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:43:22.509 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T04:43:22.509 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T04:43:22.509 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T04:43:22.510 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T04:43:22.511 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T04:43:22.511 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T04:43:22.511 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:43:22.511 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:43:22.511 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:43:22.511 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-mon.a.asok 2024-06-03T04:43:22.511 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:43:22.511 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20650/ceph-mon.a.asok config get mon_host 2024-06-03T04:43:22.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:42: TEST_profile_builtin_to_custom: run_mgr td/mclock-config x 2024-06-03T04:43:22.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-06-03T04:43:22.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T04:43:22.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T04:43:22.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T04:43:22.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-06-03T04:43:22.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T04:43:23.518 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T04:43:23.519 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:43:23.519 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:43:23.519 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:43:23.519 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:43:23.520 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:43:23.520 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20650/$cluster-$name.asok' 2024-06-03T04:43:23.520 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T04:43:23.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.20650/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T04:43:23.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:43: TEST_profile_builtin_to_custom: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-06-03T04:43:23.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-06-03T04:43:23.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T04:43:23.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T04:43:23.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T04:43:23.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-06-03T04:43:23.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=78136f38-0d6f-4bc0-8fbc-da8924e531b4 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-06-03T04:43:23.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T04:43:23.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T04:43:23.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T04:43:23.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-06-03T04:43:23.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-06-03T04:43:23.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T04:43:23.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T04:43:23.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-06-03T04:43:23.556 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T04:43:23.556 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:43:23.556 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:43:23.556 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:43:23.556 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:43:23.556 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:43:23.556 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20650/$cluster-$name.asok' 2024-06-03T04:43:23.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20650/$cluster-$name.asok' 2024-06-03T04:43:23.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T04:43:23.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T04:43:23.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T04:43:23.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-06-03T04:43:23.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-06-03T04:43:23.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T04:43:23.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T04:43:23.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T04:43:23.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T04:43:23.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T04:43:23.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-06-03T04:43:23.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-06-03T04:43:23.558 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T04:43:23.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4d01a62c-7438-477a-a883-cf4acdf18e7b 2024-06-03T04:43:23.558 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 4d01a62c-7438-477a-a883-cf4acdf18e7b 2024-06-03T04:43:23.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4d01a62c-7438-477a-a883-cf4acdf18e7b' 2024-06-03T04:43:23.558 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T04:43:23.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDrSV1m1dkWIhAApiZCyL8AlByOh6YIbUEIvA== 2024-06-03T04:43:23.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDrSV1m1dkWIhAApiZCyL8AlByOh6YIbUEIvA=="}' 2024-06-03T04:43:23.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4d01a62c-7438-477a-a883-cf4acdf18e7b -i td/mclock-config/0/new.json 2024-06-03T04:43:23.857 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T04:43:23.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-06-03T04:43:23.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=78136f38-0d6f-4bc0-8fbc-da8924e531b4 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.20650/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQDrSV1m1dkWIhAApiZCyL8AlByOh6YIbUEIvA== --osd-uuid 4d01a62c-7438-477a-a883-cf4acdf18e7b 2024-06-03T04:43:23.902 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:43:23.899+0000 7f03217b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:43:23.926 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:43:23.923+0000 7f03217b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:43:23.927 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:43:23.923+0000 7f03217b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:43:23.928 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:43:23.923+0000 7f03217b57c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-06-03T04:43:23.928 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:43:23.923+0000 7f03217b57c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-06-03T04:43:23.928 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:43:23.923+0000 7f03217b57c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-06-03T04:43:23.966 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:43:23.963+0000 7f03217b57c0 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-06-03T04:43:30.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-06-03T04:43:30.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T04:43:30.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T04:43:30.348 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T04:43:30.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T04:43:30.904 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T04:43:30.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T04:43:30.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=78136f38-0d6f-4bc0-8fbc-da8924e531b4 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.20650/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-06-03T04:43:30.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T04:43:30.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T04:43:30.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T04:43:30.934 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:43:30.931+0000 7f61fd2af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:43:30.969 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:43:30.967+0000 7f61fd2af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:43:30.970 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:43:30.967+0000 7f61fd2af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:43:31.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T04:43:31.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T04:43:31.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T04:43:31.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T04:43:31.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T04:43:31.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:43:31.320 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T04:43:31.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T04:43:31.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:43:31.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:43:31.531 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:43:31.527+0000 7f61fd2af7c0 -1 Falling back to public interface 2024-06-03T04:43:31.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:43:32.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:43:32.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:43:32.750 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T04:43:32.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T04:43:32.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:43:32.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:43:33.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:43:34.082 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:43:34.079+0000 7f61fd2af7c0 -1 osd.0 0 log_to_monitors true 2024-06-03T04:43:34.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:43:34.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:43:34.153 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T04:43:34.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T04:43:34.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:43:34.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:43:34.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:43:35.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:43:35.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:43:35.817 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T04:43:35.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T04:43:35.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:43:35.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:43:36.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:43:37.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:43:37.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:43:37.260 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T04:43:37.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T04:43:37.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:43:37.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:43:37.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:43:38.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:43:38.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:43:38.708 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T04:43:38.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T04:43:38.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:43:38.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:43:39.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:43:40.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:43:40.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:43:40.270 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T04:43:40.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T04:43:40.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:43:40.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:43:40.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:43:41.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:43:41.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:43:41.709 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T04:43:41.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T04:43:41.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:43:41.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:43:42.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:43:43.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:43:43.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:43:43.363 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T04:43:43.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T04:43:43.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:43:43.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:43:43.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:43:44.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:43:44.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:43:44.892 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T04:43:44.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T04:43:44.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:43:44.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:43:45.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:43:46.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:43:46.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:43:46.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T04:43:46.340 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T04:43:46.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:43:46.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:43:46.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:43:47.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:43:47.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:43:47.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T04:43:47.747 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T04:43:47.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:43:47.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:43:48.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:43:49.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:43:49.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:43:49.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T04:43:49.352 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T04:43:49.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:43:49.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:43:50.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:43:51.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:43:51.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:43:51.051 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T04:43:51.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T04:43:51.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:43:51.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:43:51.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:43:52.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:43:52.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:43:52.740 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T04:43:52.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T04:43:52.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:43:52.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:43:53.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:43:54.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:43:54.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:43:54.496 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T04:43:54.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T04:43:54.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:43:54.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:43:54.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:43:55.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:43:55.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:43:55.968 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T04:43:55.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T04:43:55.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:43:55.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:43:57.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:43:58.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:43:58.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:43:58.072 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T04:43:58.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T04:43:58.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:43:58.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:43:58.595 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3146318223,v1:127.0.0.1:6803/3146318223] [v2:127.0.0.1:6804/3146318223,v1:127.0.0.1:6805/3146318223] exists,up 4d01a62c-7438-477a-a883-cf4acdf18e7b 2024-06-03T04:43:58.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T04:43:58.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T04:43:58.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T04:43:58.596 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:46: TEST_profile_builtin_to_custom: ceph config get osd.0 osd_mclock_profile 2024-06-03T04:43:59.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:46: TEST_profile_builtin_to_custom: local mclock_profile=balanced 2024-06-03T04:43:59.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:47: TEST_profile_builtin_to_custom: test balanced = balanced 2024-06-03T04:43:59.009 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-06-03T04:43:59.009 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:43:59.009 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: jq .osd_mclock_profile 2024-06-03T04:43:59.009 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:43:59.010 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:43:59.010 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:43:59.010 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:43:59.010 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-osd.0.asok 2024-06-03T04:43:59.011 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-06-03T04:43:59.011 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.20650/ceph-osd.0.asok config get osd_mclock_profile 2024-06-03T04:43:59.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: mclock_profile='"high_recovery_ops"' 2024-06-03T04:43:59.133 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: eval echo '"high_recovery_ops"' 2024-06-03T04:43:59.133 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: echo high_recovery_ops 2024-06-03T04:43:59.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: mclock_profile=high_recovery_ops 2024-06-03T04:43:59.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:54: TEST_profile_builtin_to_custom: test high_recovery_ops = high_recovery_ops 2024-06-03T04:43:59.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:57: TEST_profile_builtin_to_custom: ceph tell osd.0 config set osd_mclock_profile custom 2024-06-03T04:43:59.316 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-06-03T04:43:59.316 INFO:tasks.workunit.client.0.smithi088.stdout: "success": "" 2024-06-03T04:43:59.317 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-06-03T04:43:59.334 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: jq .osd_mclock_profile 2024-06-03T04:43:59.335 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-06-03T04:43:59.335 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:43:59.335 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:43:59.335 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:43:59.335 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:43:59.335 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:43:59.335 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-osd.0.asok 2024-06-03T04:43:59.335 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-06-03T04:43:59.335 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.20650/ceph-osd.0.asok config get osd_mclock_profile 2024-06-03T04:43:59.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: mclock_profile='"custom"' 2024-06-03T04:43:59.456 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: eval echo '"custom"' 2024-06-03T04:43:59.456 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: echo custom 2024-06-03T04:43:59.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: mclock_profile=custom 2024-06-03T04:43:59.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:63: TEST_profile_builtin_to_custom: test custom = custom 2024-06-03T04:43:59.457 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: jq .osd_mclock_scheduler_client_res 2024-06-03T04:43:59.458 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: bc 2024-06-03T04:43:59.458 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-06-03T04:43:59.458 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:43:59.458 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:43:59.459 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:43:59.459 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:43:59.459 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:43:59.459 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-osd.0.asok 2024-06-03T04:43:59.459 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-06-03T04:43:59.459 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.20650/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-06-03T04:43:59.575 INFO:tasks.workunit.client.0.smithi088.stdout:client_res = 0.300000 2024-06-03T04:43:59.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: local client_res=0.300000 2024-06-03T04:43:59.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:69: TEST_profile_builtin_to_custom: echo 'client_res = 0.300000' 2024-06-03T04:43:59.576 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: bc -l 2024-06-03T04:43:59.576 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: echo '0.300000 + 0.1' 2024-06-03T04:43:59.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: local client_res_new=.400000 2024-06-03T04:43:59.577 INFO:tasks.workunit.client.0.smithi088.stdout:client_res_new = .400000 2024-06-03T04:43:59.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:71: TEST_profile_builtin_to_custom: echo 'client_res_new = .400000' 2024-06-03T04:43:59.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:72: TEST_profile_builtin_to_custom: ceph config set osd.0 osd_mclock_scheduler_client_res .400000 2024-06-03T04:44:00.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:77: TEST_profile_builtin_to_custom: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-06-03T04:44:00.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:77: TEST_profile_builtin_to_custom: local res=0.400000 2024-06-03T04:44:00.487 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: echo '0.400000 != .400000' 2024-06-03T04:44:00.487 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: bc -l 2024-06-03T04:44:00.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: (( 0 )) 2024-06-03T04:44:00.490 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: jq .osd_mclock_scheduler_client_res 2024-06-03T04:44:00.490 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-06-03T04:44:00.490 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:44:00.490 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:44:00.490 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: bc 2024-06-03T04:44:00.491 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:44:00.491 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:44:00.491 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:44:00.491 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-osd.0.asok 2024-06-03T04:44:00.491 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-06-03T04:44:00.491 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.20650/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-06-03T04:44:00.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: res=0.400000 2024-06-03T04:44:00.605 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: echo '0.400000 != .400000' 2024-06-03T04:44:00.605 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: bc -l 2024-06-03T04:44:00.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: (( 0 )) 2024-06-03T04:44:00.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:89: TEST_profile_builtin_to_custom: teardown td/mclock-config 2024-06-03T04:44:00.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-03T04:44:00.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T04:44:00.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-03T04:44:00.609 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:44:00.609 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:44:00.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:44:00.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:44:00.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:44:00.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:44:00.715 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:44:00.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:44:00.716 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:44:00.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:44:00.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:44:00.717 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:44:00.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:44:00.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:44:00.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:44:00.719 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:44:00.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:44:00.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T04:44:00.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-03T04:44:00.725 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:44:00.726 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:44:00.726 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:44:00.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20650 2024-06-03T04:44:00.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:44:00.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:44:00.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-06-03T04:44:00.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-03T04:44:00.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T04:44:00.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-03T04:44:00.727 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:44:00.728 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:44:00.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:44:00.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:44:00.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:44:00.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:44:00.729 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:44:00.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:44:00.730 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:44:00.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:44:00.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:44:00.732 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:44:00.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:44:00.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:44:00.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:44:00.733 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:44:00.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:44:00.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T04:44:00.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-03T04:44:00.736 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:44:00.736 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:44:00.736 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:44:00.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20650 2024-06-03T04:44:00.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:44:00.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:44:00.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-06-03T04:44:00.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-06-03T04:44:00.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-06-03T04:44:00.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-06-03T04:44:00.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-03T04:44:00.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T04:44:00.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-03T04:44:00.738 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:44:00.738 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:44:00.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:44:00.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:44:00.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:44:00.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:44:00.739 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:44:00.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:44:00.741 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:44:00.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:44:00.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:44:00.742 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:44:00.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:44:00.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:44:00.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:44:00.744 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:44:00.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:44:00.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T04:44:00.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-03T04:44:00.746 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:44:00.746 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:44:00.746 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:44:00.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20650 2024-06-03T04:44:00.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:44:00.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:44:00.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-06-03T04:44:00.748 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T04:44:00.748 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:44:00.749 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:44:00.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20650 2024-06-03T04:44:00.751 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T04:44:00.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T04:44:00.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T04:44:00.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-06-03T04:44:00.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_custom_to_builtin td/mclock-config 2024-06-03T04:44:00.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:93: TEST_profile_custom_to_builtin: local dir=td/mclock-config 2024-06-03T04:44:00.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:95: TEST_profile_custom_to_builtin: setup td/mclock-config 2024-06-03T04:44:00.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-06-03T04:44:00.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-06-03T04:44:00.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-03T04:44:00.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T04:44:00.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-03T04:44:00.752 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:44:00.752 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:44:00.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:44:00.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:44:00.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:44:00.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:44:00.753 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:44:00.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:44:00.754 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:44:00.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:44:00.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:44:00.755 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:44:00.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:44:00.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:44:00.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:44:00.756 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:44:00.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:44:00.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T04:44:00.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-03T04:44:00.759 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:44:00.759 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:44:00.759 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:44:00.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20650 2024-06-03T04:44:00.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:44:00.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:44:00.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-06-03T04:44:00.761 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T04:44:00.761 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:44:00.761 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:44:00.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20650 2024-06-03T04:44:00.763 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T04:44:00.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T04:44:00.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T04:44:00.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-06-03T04:44:00.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:96: TEST_profile_custom_to_builtin: run_mon td/mclock-config a 2024-06-03T04:44:00.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-06-03T04:44:00.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T04:44:00.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T04:44:00.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T04:44:00.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-06-03T04:44:00.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-06-03T04:44:01.168 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T04:44:01.168 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:44:01.168 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:44:01.169 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:44:01.169 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:44:01.169 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:44:01.169 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20650/$cluster-$name.asok' 2024-06-03T04:44:01.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.20650/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-03T04:44:01.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T04:44:01.364 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T04:44:01.364 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:44:01.364 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T04:44:01.364 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T04:44:01.365 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T04:44:01.366 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T04:44:01.366 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T04:44:01.366 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T04:44:01.366 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:44:01.366 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:44:01.366 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:44:01.366 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-mon.a.asok 2024-06-03T04:44:01.367 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:44:01.367 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20650/ceph-mon.a.asok config get fsid 2024-06-03T04:44:01.484 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T04:44:01.484 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:44:01.484 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T04:44:01.484 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T04:44:01.484 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T04:44:01.484 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T04:44:01.484 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T04:44:01.484 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T04:44:01.485 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:44:01.485 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:44:01.485 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:44:01.485 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-mon.a.asok 2024-06-03T04:44:01.485 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:44:01.485 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20650/ceph-mon.a.asok config get mon_host 2024-06-03T04:44:01.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:97: TEST_profile_custom_to_builtin: run_mgr td/mclock-config x 2024-06-03T04:44:01.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-06-03T04:44:01.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T04:44:01.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T04:44:01.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T04:44:01.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-06-03T04:44:01.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T04:44:02.508 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T04:44:02.553 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:44:02.553 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:44:02.557 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:44:02.557 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:44:02.557 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:44:02.557 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20650/$cluster-$name.asok' 2024-06-03T04:44:02.557 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T04:44:02.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.20650/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T04:44:02.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:98: TEST_profile_custom_to_builtin: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-06-03T04:44:02.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-06-03T04:44:02.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T04:44:02.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T04:44:02.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T04:44:02.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-06-03T04:44:02.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=78136f38-0d6f-4bc0-8fbc-da8924e531b4 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-06-03T04:44:02.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T04:44:02.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T04:44:02.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T04:44:02.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-06-03T04:44:02.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-06-03T04:44:02.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T04:44:02.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T04:44:02.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-06-03T04:44:02.558 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T04:44:02.559 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:44:02.559 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:44:02.559 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:44:02.559 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:44:02.559 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:44:02.559 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20650/$cluster-$name.asok' 2024-06-03T04:44:02.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20650/$cluster-$name.asok' 2024-06-03T04:44:02.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T04:44:02.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T04:44:02.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T04:44:02.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-06-03T04:44:02.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-06-03T04:44:02.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T04:44:02.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T04:44:02.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T04:44:02.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T04:44:02.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T04:44:02.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-06-03T04:44:02.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-06-03T04:44:02.560 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T04:44:02.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=02125ddc-809d-424b-88bc-964f1c6170eb 2024-06-03T04:44:02.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 02125ddc-809d-424b-88bc-964f1c6170eb' 2024-06-03T04:44:02.560 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 02125ddc-809d-424b-88bc-964f1c6170eb 2024-06-03T04:44:02.561 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T04:44:02.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQASSl1mgmGhIRAAcEngU4DrFvM4xcfUIvC3uQ== 2024-06-03T04:44:02.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQASSl1mgmGhIRAAcEngU4DrFvM4xcfUIvC3uQ=="}' 2024-06-03T04:44:02.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 02125ddc-809d-424b-88bc-964f1c6170eb -i td/mclock-config/0/new.json 2024-06-03T04:44:02.900 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T04:44:02.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-06-03T04:44:02.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=78136f38-0d6f-4bc0-8fbc-da8924e531b4 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.20650/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQASSl1mgmGhIRAAcEngU4DrFvM4xcfUIvC3uQ== --osd-uuid 02125ddc-809d-424b-88bc-964f1c6170eb 2024-06-03T04:44:02.946 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:44:02.943+0000 7fc96ac587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:44:02.978 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:44:02.975+0000 7fc96ac587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:44:02.979 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:44:02.975+0000 7fc96ac587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:44:02.979 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:44:02.975+0000 7fc96ac587c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-06-03T04:44:02.980 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:44:02.975+0000 7fc96ac587c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-06-03T04:44:02.980 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:44:02.975+0000 7fc96ac587c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-06-03T04:44:03.017 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:44:03.015+0000 7fc96ac587c0 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-06-03T04:44:09.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-06-03T04:44:09.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T04:44:09.355 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T04:44:09.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T04:44:09.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T04:44:09.893 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T04:44:09.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T04:44:09.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T04:44:09.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T04:44:09.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=78136f38-0d6f-4bc0-8fbc-da8924e531b4 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.20650/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-06-03T04:44:09.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T04:44:09.925 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:44:09.922+0000 7f071690e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:44:09.959 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:44:09.954+0000 7f071690e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:44:09.961 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:44:09.958+0000 7f071690e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:44:10.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T04:44:10.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T04:44:10.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T04:44:10.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T04:44:10.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T04:44:10.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:44:10.341 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T04:44:10.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T04:44:10.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:44:10.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:44:10.527 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:44:10.522+0000 7f071690e7c0 -1 Falling back to public interface 2024-06-03T04:44:10.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:44:11.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:44:11.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:44:11.764 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T04:44:11.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T04:44:11.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:44:11.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:44:12.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:44:13.175 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:44:13.170+0000 7f071690e7c0 -1 osd.0 0 log_to_monitors true 2024-06-03T04:44:13.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:44:13.182 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T04:44:13.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:44:13.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T04:44:13.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:44:13.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:44:13.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:44:14.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:44:14.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:44:14.697 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T04:44:14.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T04:44:14.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:44:14.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:44:15.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:44:16.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:44:16.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:44:16.174 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T04:44:16.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T04:44:16.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:44:16.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:44:16.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:44:17.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:44:17.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:44:17.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T04:44:17.587 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T04:44:17.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:44:17.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:44:18.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:44:19.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:44:19.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:44:19.136 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T04:44:19.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T04:44:19.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:44:19.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:44:19.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:44:20.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:44:20.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:44:20.558 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T04:44:20.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T04:44:20.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:44:20.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:44:21.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:44:22.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:44:22.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:44:22.060 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T04:44:22.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T04:44:22.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:44:22.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:44:22.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:44:23.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:44:23.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:44:23.498 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T04:44:23.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T04:44:23.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:44:23.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:44:23.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:44:24.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:44:24.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:44:24.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T04:44:24.993 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T04:44:24.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:44:24.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:44:25.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:44:26.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:44:26.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:44:26.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T04:44:26.408 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T04:44:26.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:44:26.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:44:26.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:44:27.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:44:27.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:44:27.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T04:44:27.825 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T04:44:27.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:44:27.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:44:28.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:44:29.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:44:29.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:44:29.653 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T04:44:29.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T04:44:29.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:44:29.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:44:30.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:44:31.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:44:31.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:44:31.361 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T04:44:31.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T04:44:31.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:44:31.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:44:32.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:44:33.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:44:33.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:44:33.477 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T04:44:33.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T04:44:33.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:44:33.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:44:34.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:44:35.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:44:35.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:44:35.708 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T04:44:35.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T04:44:35.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:44:35.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:44:36.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:44:37.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:44:37.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:44:37.914 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T04:44:37.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T04:44:37.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:44:37.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:44:38.354 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3084361469,v1:127.0.0.1:6803/3084361469] [v2:127.0.0.1:6804/3084361469,v1:127.0.0.1:6805/3084361469] exists,up 02125ddc-809d-424b-88bc-964f1c6170eb 2024-06-03T04:44:38.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T04:44:38.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T04:44:38.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T04:44:38.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:101: TEST_profile_custom_to_builtin: local def_mclock_profile 2024-06-03T04:44:38.355 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:102: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_profile 2024-06-03T04:44:38.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:102: TEST_profile_custom_to_builtin: def_mclock_profile=balanced 2024-06-03T04:44:38.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:103: TEST_profile_custom_to_builtin: test balanced = balanced 2024-06-03T04:44:38.753 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2024-06-03T04:44:38.754 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-06-03T04:44:38.754 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:44:38.754 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:44:38.755 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:44:38.755 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:44:38.755 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:44:38.755 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-osd.0.asok 2024-06-03T04:44:38.755 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-06-03T04:44:38.755 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.20650/ceph-osd.0.asok config get osd_mclock_profile 2024-06-03T04:44:38.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: local 'orig_mclock_profile="high_recovery_ops"' 2024-06-03T04:44:38.871 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: eval echo '"high_recovery_ops"' 2024-06-03T04:44:38.871 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: echo high_recovery_ops 2024-06-03T04:44:38.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: orig_mclock_profile=high_recovery_ops 2024-06-03T04:44:38.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:110: TEST_profile_custom_to_builtin: test high_recovery_ops = high_recovery_ops 2024-06-03T04:44:38.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:113: TEST_profile_custom_to_builtin: ceph tell osd.0 config set osd_mclock_profile custom 2024-06-03T04:44:39.004 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-06-03T04:44:39.004 INFO:tasks.workunit.client.0.smithi088.stdout: "success": "" 2024-06-03T04:44:39.004 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-06-03T04:44:39.018 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2024-06-03T04:44:39.018 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-06-03T04:44:39.018 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:44:39.018 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:44:39.018 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:44:39.018 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:44:39.019 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:44:39.019 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-osd.0.asok 2024-06-03T04:44:39.019 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-06-03T04:44:39.019 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.20650/ceph-osd.0.asok config get osd_mclock_profile 2024-06-03T04:44:39.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: local 'mclock_profile="custom"' 2024-06-03T04:44:39.132 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: eval echo '"custom"' 2024-06-03T04:44:39.132 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: echo custom 2024-06-03T04:44:39.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: mclock_profile=custom 2024-06-03T04:44:39.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:120: TEST_profile_custom_to_builtin: test custom = custom 2024-06-03T04:44:39.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:123: TEST_profile_custom_to_builtin: local client_res 2024-06-03T04:44:39.133 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-06-03T04:44:39.133 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: bc 2024-06-03T04:44:39.134 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-06-03T04:44:39.135 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:44:39.135 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:44:39.135 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:44:39.135 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:44:39.135 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:44:39.135 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-osd.0.asok 2024-06-03T04:44:39.135 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-06-03T04:44:39.136 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.20650/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-06-03T04:44:39.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: client_res=0.300000 2024-06-03T04:44:39.244 INFO:tasks.workunit.client.0.smithi088.stdout:Original client_res for osd.0 = 0.300000 2024-06-03T04:44:39.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:127: TEST_profile_custom_to_builtin: echo 'Original client_res for osd.0 = 0.300000' 2024-06-03T04:44:39.245 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: echo '0.300000 + 0.1' 2024-06-03T04:44:39.245 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: bc -l 2024-06-03T04:44:39.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: local client_res_new=.400000 2024-06-03T04:44:39.246 INFO:tasks.workunit.client.0.smithi088.stdout:client_res_new = .400000 2024-06-03T04:44:39.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:131: TEST_profile_custom_to_builtin: echo 'client_res_new = .400000' 2024-06-03T04:44:39.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:132: TEST_profile_custom_to_builtin: ceph config set osd osd_mclock_scheduler_client_res .400000 2024-06-03T04:44:39.728 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:136: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-06-03T04:44:40.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:136: TEST_profile_custom_to_builtin: local res=0.400000 2024-06-03T04:44:40.117 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-06-03T04:44:40.117 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: bc -l 2024-06-03T04:44:40.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: (( 0 )) 2024-06-03T04:44:40.121 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: bc 2024-06-03T04:44:40.121 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-06-03T04:44:40.121 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:44:40.122 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:44:40.122 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-06-03T04:44:40.122 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:44:40.122 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:44:40.122 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:44:40.122 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-osd.0.asok 2024-06-03T04:44:40.122 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-06-03T04:44:40.122 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.20650/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-06-03T04:44:40.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: res=0.400000 2024-06-03T04:44:40.235 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: bc -l 2024-06-03T04:44:40.235 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-06-03T04:44:40.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: (( 0 )) 2024-06-03T04:44:40.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:154: TEST_profile_custom_to_builtin: ceph tell osd.0 config set osd_mclock_profile high_recovery_ops 2024-06-03T04:44:40.369 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-06-03T04:44:40.369 INFO:tasks.workunit.client.0.smithi088.stdout: "success": "" 2024-06-03T04:44:40.370 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-06-03T04:44:40.384 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2024-06-03T04:44:40.384 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-06-03T04:44:40.384 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:44:40.384 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:44:40.384 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:44:40.384 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:44:40.384 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:44:40.385 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-osd.0.asok 2024-06-03T04:44:40.385 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-06-03T04:44:40.385 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.20650/ceph-osd.0.asok config get osd_mclock_profile 2024-06-03T04:44:40.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: eval 'mclock_profile="high_recovery_ops"' 2024-06-03T04:44:40.491 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: mclock_profile=high_recovery_ops 2024-06-03T04:44:40.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:160: TEST_profile_custom_to_builtin: test high_recovery_ops = high_recovery_ops 2024-06-03T04:44:40.491 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:165: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-06-03T04:44:40.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:165: TEST_profile_custom_to_builtin: local res=0.400000 2024-06-03T04:44:40.907 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-06-03T04:44:40.907 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: bc -l 2024-06-03T04:44:40.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: (( 0 )) 2024-06-03T04:44:40.910 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-06-03T04:44:40.910 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: bc 2024-06-03T04:44:40.911 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-06-03T04:44:40.911 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:44:40.911 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:44:40.911 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:44:40.912 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:44:40.912 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:44:40.912 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-osd.0.asok 2024-06-03T04:44:40.912 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-06-03T04:44:40.912 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.20650/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-06-03T04:44:41.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: res=0.400000 2024-06-03T04:44:41.018 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-06-03T04:44:41.018 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: bc -l 2024-06-03T04:44:41.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: (( 0 )) 2024-06-03T04:44:41.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:178: TEST_profile_custom_to_builtin: ceph config rm osd osd_mclock_scheduler_client_res 2024-06-03T04:44:41.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:180: TEST_profile_custom_to_builtin: sleep 5 2024-06-03T04:44:46.547 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:185: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-06-03T04:44:46.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:185: TEST_profile_custom_to_builtin: res=0.000000 2024-06-03T04:44:46.951 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: echo '0.000000 != 0.0' 2024-06-03T04:44:46.951 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: bc -l 2024-06-03T04:44:46.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: (( 0 )) 2024-06-03T04:44:46.954 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-06-03T04:44:46.954 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: bc 2024-06-03T04:44:46.954 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-06-03T04:44:46.954 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:44:46.954 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:44:46.955 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:44:46.955 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:44:46.955 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:44:46.955 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-osd.0.asok 2024-06-03T04:44:46.955 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-06-03T04:44:46.955 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.20650/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-06-03T04:44:47.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: res=0.300000 2024-06-03T04:44:47.061 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: echo '0.300000 != 0.300000' 2024-06-03T04:44:47.061 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: bc -l 2024-06-03T04:44:47.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: (( 0 )) 2024-06-03T04:44:47.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:198: TEST_profile_custom_to_builtin: teardown td/mclock-config 2024-06-03T04:44:47.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-03T04:44:47.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T04:44:47.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-03T04:44:47.064 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:44:47.064 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:44:47.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:44:47.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:44:47.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:44:47.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:44:47.372 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:44:47.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:44:47.374 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:44:47.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:44:47.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:44:47.376 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:44:47.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:44:47.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:44:47.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:44:47.379 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:44:47.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:44:47.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T04:44:47.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-03T04:44:47.388 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:44:47.388 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:44:47.389 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:44:47.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20650 2024-06-03T04:44:47.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:44:47.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:44:47.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-06-03T04:44:47.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-03T04:44:47.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T04:44:47.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-03T04:44:47.391 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:44:47.391 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:44:47.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:44:47.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:44:47.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:44:47.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:44:47.394 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:44:47.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:44:47.395 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:44:47.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:44:47.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:44:47.397 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:44:47.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:44:47.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:44:47.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:44:47.398 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:44:47.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:44:47.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T04:44:47.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-03T04:44:47.402 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:44:47.402 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:44:47.402 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:44:47.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20650 2024-06-03T04:44:47.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:44:47.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:44:47.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-06-03T04:44:47.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-06-03T04:44:47.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-06-03T04:44:47.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-06-03T04:44:47.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-03T04:44:47.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T04:44:47.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-03T04:44:47.405 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:44:47.405 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:44:47.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:44:47.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:44:47.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:44:47.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:44:47.407 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:44:47.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:44:47.409 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:44:47.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:44:47.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:44:47.410 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:44:47.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:44:47.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:44:47.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:44:47.412 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:44:47.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:44:47.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T04:44:47.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-03T04:44:47.416 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:44:47.416 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:44:47.416 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:44:47.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20650 2024-06-03T04:44:47.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:44:47.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:44:47.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-06-03T04:44:47.419 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T04:44:47.419 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:44:47.419 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:44:47.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20650 2024-06-03T04:44:47.420 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T04:44:47.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T04:44:47.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T04:44:47.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-06-03T04:44:47.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_disallow_builtin_params_modify td/mclock-config 2024-06-03T04:44:47.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:305: TEST_profile_disallow_builtin_params_modify: local dir=td/mclock-config 2024-06-03T04:44:47.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:307: TEST_profile_disallow_builtin_params_modify: setup td/mclock-config 2024-06-03T04:44:47.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-06-03T04:44:47.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-06-03T04:44:47.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-03T04:44:47.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T04:44:47.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-03T04:44:47.423 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:44:47.423 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:44:47.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:44:47.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:44:47.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:44:47.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:44:47.424 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:44:47.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:44:47.426 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:44:47.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:44:47.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:44:47.427 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:44:47.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:44:47.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:44:47.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:44:47.429 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:44:47.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:44:47.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T04:44:47.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-03T04:44:47.433 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:44:47.433 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:44:47.433 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:44:47.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20650 2024-06-03T04:44:47.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:44:47.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:44:47.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-06-03T04:44:47.436 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T04:44:47.436 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:44:47.436 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:44:47.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20650 2024-06-03T04:44:47.438 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T04:44:47.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T04:44:47.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T04:44:47.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-06-03T04:44:47.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:308: TEST_profile_disallow_builtin_params_modify: run_mon td/mclock-config a 2024-06-03T04:44:47.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-06-03T04:44:47.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T04:44:47.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T04:44:47.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T04:44:47.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-06-03T04:44:47.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-06-03T04:44:47.823 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T04:44:47.823 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:44:47.823 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:44:47.824 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:44:47.824 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:44:47.824 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:44:47.824 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20650/$cluster-$name.asok' 2024-06-03T04:44:47.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.20650/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-03T04:44:48.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T04:44:48.034 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T04:44:48.034 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:44:48.035 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T04:44:48.035 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T04:44:48.035 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T04:44:48.035 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T04:44:48.035 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T04:44:48.036 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T04:44:48.036 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:44:48.036 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:44:48.036 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:44:48.036 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-mon.a.asok 2024-06-03T04:44:48.037 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:44:48.037 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20650/ceph-mon.a.asok config get fsid 2024-06-03T04:44:48.152 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T04:44:48.152 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:44:48.152 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T04:44:48.152 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T04:44:48.153 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T04:44:48.153 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T04:44:48.153 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T04:44:48.154 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T04:44:48.154 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:44:48.154 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:44:48.154 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:44:48.154 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-mon.a.asok 2024-06-03T04:44:48.154 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:44:48.154 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20650/ceph-mon.a.asok config get mon_host 2024-06-03T04:44:48.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:309: TEST_profile_disallow_builtin_params_modify: run_mgr td/mclock-config x 2024-06-03T04:44:48.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-06-03T04:44:48.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T04:44:48.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T04:44:48.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T04:44:48.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-06-03T04:44:48.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T04:44:49.174 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T04:44:49.174 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:44:49.174 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:44:49.174 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:44:49.174 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:44:49.174 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:44:49.175 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20650/$cluster-$name.asok' 2024-06-03T04:44:49.175 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T04:44:49.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.20650/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T04:44:49.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:311: TEST_profile_disallow_builtin_params_modify: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-06-03T04:44:49.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-06-03T04:44:49.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T04:44:49.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T04:44:49.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T04:44:49.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-06-03T04:44:49.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=78136f38-0d6f-4bc0-8fbc-da8924e531b4 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-06-03T04:44:49.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T04:44:49.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T04:44:49.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T04:44:49.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-06-03T04:44:49.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-06-03T04:44:49.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T04:44:49.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T04:44:49.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-06-03T04:44:49.211 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T04:44:49.211 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:44:49.211 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:44:49.211 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:44:49.211 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:44:49.211 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:44:49.211 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20650/$cluster-$name.asok' 2024-06-03T04:44:49.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20650/$cluster-$name.asok' 2024-06-03T04:44:49.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T04:44:49.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T04:44:49.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T04:44:49.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-06-03T04:44:49.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-06-03T04:44:49.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T04:44:49.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T04:44:49.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T04:44:49.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T04:44:49.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T04:44:49.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-06-03T04:44:49.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-06-03T04:44:49.212 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T04:44:49.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b7a25377-e438-4075-9fb7-6f3a0f2abff1 2024-06-03T04:44:49.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b7a25377-e438-4075-9fb7-6f3a0f2abff1' 2024-06-03T04:44:49.213 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 b7a25377-e438-4075-9fb7-6f3a0f2abff1 2024-06-03T04:44:49.213 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T04:44:49.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBBSl1mPUB1DRAAmqJ6Kbx/Bq8jRE8mL7YmcA== 2024-06-03T04:44:49.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBBSl1mPUB1DRAAmqJ6Kbx/Bq8jRE8mL7YmcA=="}' 2024-06-03T04:44:49.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b7a25377-e438-4075-9fb7-6f3a0f2abff1 -i td/mclock-config/0/new.json 2024-06-03T04:44:49.548 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T04:44:49.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-06-03T04:44:49.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=78136f38-0d6f-4bc0-8fbc-da8924e531b4 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.20650/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQBBSl1mPUB1DRAAmqJ6Kbx/Bq8jRE8mL7YmcA== --osd-uuid b7a25377-e438-4075-9fb7-6f3a0f2abff1 2024-06-03T04:44:49.596 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:44:49.590+0000 7f07d8eed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:44:49.624 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:44:49.618+0000 7f07d8eed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:44:49.626 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:44:49.622+0000 7f07d8eed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:44:49.626 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:44:49.622+0000 7f07d8eed7c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-06-03T04:44:49.626 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:44:49.622+0000 7f07d8eed7c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-06-03T04:44:49.626 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:44:49.622+0000 7f07d8eed7c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-06-03T04:44:49.664 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:44:49.658+0000 7f07d8eed7c0 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-06-03T04:44:56.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-06-03T04:44:56.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T04:44:56.022 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T04:44:56.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T04:44:56.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T04:44:56.568 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T04:44:56.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T04:44:56.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=78136f38-0d6f-4bc0-8fbc-da8924e531b4 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.20650/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-06-03T04:44:56.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T04:44:56.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T04:44:56.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T04:44:56.599 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:44:56.594+0000 7f923058c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:44:56.631 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:44:56.626+0000 7f923058c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:44:56.633 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:44:56.630+0000 7f923058c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:44:57.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T04:44:57.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T04:44:57.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T04:44:57.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T04:44:57.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T04:44:57.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:44:57.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T04:44:57.013 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T04:44:57.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:44:57.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:44:57.222 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:44:57.218+0000 7f923058c7c0 -1 Falling back to public interface 2024-06-03T04:44:57.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:44:58.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:44:58.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:44:58.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T04:44:58.454 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T04:44:58.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:44:58.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:44:58.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:44:59.802 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:44:59.798+0000 7f923058c7c0 -1 osd.0 0 log_to_monitors true 2024-06-03T04:44:59.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:44:59.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:44:59.852 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T04:44:59.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T04:44:59.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:44:59.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:45:00.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:45:01.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:45:01.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:45:01.499 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T04:45:01.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T04:45:01.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:45:01.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:45:01.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:45:02.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:45:02.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:45:02.912 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T04:45:02.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T04:45:02.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:45:02.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:45:03.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:45:04.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:45:04.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:45:04.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T04:45:04.336 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T04:45:04.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:45:04.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:45:04.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:45:05.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:45:05.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:45:05.811 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T04:45:05.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T04:45:05.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:45:05.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:45:06.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:45:07.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:45:07.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:45:07.227 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T04:45:07.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T04:45:07.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:45:07.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:45:07.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:45:08.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:45:08.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:45:08.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T04:45:08.762 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T04:45:08.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:45:08.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:45:09.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:45:10.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:45:10.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:45:10.241 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T04:45:10.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T04:45:10.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:45:10.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:45:10.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:45:11.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:45:11.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:45:11.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T04:45:11.747 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T04:45:11.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:45:11.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:45:12.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:45:13.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:45:13.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:45:13.187 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T04:45:13.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T04:45:13.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:45:13.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:45:13.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:45:14.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:45:14.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:45:14.718 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T04:45:14.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T04:45:14.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:45:14.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:45:15.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:45:16.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:45:16.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:45:16.554 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T04:45:16.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T04:45:16.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:45:16.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:45:17.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:45:18.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:45:18.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:45:18.399 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T04:45:18.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T04:45:18.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:45:18.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:45:19.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:45:20.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:45:20.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:45:20.112 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T04:45:20.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T04:45:20.113 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:45:20.113 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:45:21.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:45:22.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:45:22.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:45:22.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T04:45:22.077 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T04:45:22.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:45:22.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:45:22.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:45:23.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:45:23.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:45:23.917 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T04:45:23.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T04:45:23.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:45:23.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:45:24.341 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1700118078,v1:127.0.0.1:6803/1700118078] [v2:127.0.0.1:6804/1700118078,v1:127.0.0.1:6805/1700118078] exists,up b7a25377-e438-4075-9fb7-6f3a0f2abff1 2024-06-03T04:45:24.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T04:45:24.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T04:45:24.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T04:45:24.342 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:314: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_profile 2024-06-03T04:45:24.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:314: TEST_profile_disallow_builtin_params_modify: local def_mclock_profile=balanced 2024-06-03T04:45:24.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:315: TEST_profile_disallow_builtin_params_modify: test balanced = balanced 2024-06-03T04:45:24.795 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_profile 2024-06-03T04:45:24.795 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-06-03T04:45:24.796 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:45:24.796 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:45:24.797 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:45:24.797 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:45:24.797 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:45:24.797 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-osd.0.asok 2024-06-03T04:45:24.797 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-06-03T04:45:24.797 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.20650/ceph-osd.0.asok config get osd_mclock_profile 2024-06-03T04:45:24.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: local 'cur_mclock_profile="high_recovery_ops"' 2024-06-03T04:45:24.918 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: eval echo '"high_recovery_ops"' 2024-06-03T04:45:24.918 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: echo high_recovery_ops 2024-06-03T04:45:24.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: cur_mclock_profile=high_recovery_ops 2024-06-03T04:45:24.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:322: TEST_profile_disallow_builtin_params_modify: test high_recovery_ops = high_recovery_ops 2024-06-03T04:45:24.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:324: TEST_profile_disallow_builtin_params_modify: options=('osd_mclock_scheduler_background_recovery_res' 'osd_mclock_scheduler_client_res') 2024-06-03T04:45:24.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:324: TEST_profile_disallow_builtin_params_modify: declare -a options 2024-06-03T04:45:24.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:327: TEST_profile_disallow_builtin_params_modify: local retries=10 2024-06-03T04:45:24.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:328: TEST_profile_disallow_builtin_params_modify: local errors=0 2024-06-03T04:45:24.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:329: TEST_profile_disallow_builtin_params_modify: for opt in "${options[@]}" 2024-06-03T04:45:24.920 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_background_recovery_res 2024-06-03T04:45:24.920 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: bc 2024-06-03T04:45:24.920 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-06-03T04:45:24.920 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:45:24.920 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:45:24.920 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:45:24.920 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:45:24.920 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:45:24.920 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-osd.0.asok 2024-06-03T04:45:24.920 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-06-03T04:45:24.921 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.20650/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-06-03T04:45:25.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: local opt_val_orig=0.700000 2024-06-03T04:45:25.034 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: echo '0.700000 + 0.1' 2024-06-03T04:45:25.035 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: bc -l 2024-06-03T04:45:25.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: local opt_val_new=.800000 2024-06-03T04:45:25.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:335: TEST_profile_disallow_builtin_params_modify: ceph config set osd.0 osd_mclock_scheduler_background_recovery_res .800000 2024-06-03T04:45:25.575 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: expr 10 - 1 2024-06-03T04:45:25.576 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: seq 0 9 2024-06-03T04:45:25.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: for count in $(seq 0 $(expr $retries - 1)) 2024-06-03T04:45:25.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:340: TEST_profile_disallow_builtin_params_modify: errors=0 2024-06-03T04:45:25.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:341: TEST_profile_disallow_builtin_params_modify: sleep 2 2024-06-03T04:45:27.616 INFO:tasks.workunit.client.0.smithi088.stdout:Check configuration values - Attempt#: 0 2024-06-03T04:45:27.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:343: TEST_profile_disallow_builtin_params_modify: echo 'Check configuration values - Attempt#: 0' 2024-06-03T04:45:27.617 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_scheduler_background_recovery_res 2024-06-03T04:45:28.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: local res=0.000000 2024-06-03T04:45:28.019 INFO:tasks.workunit.client.0.smithi088.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000 2024-06-03T04:45:28.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:346: TEST_profile_disallow_builtin_params_modify: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000' 2024-06-03T04:45:28.019 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: bc -l 2024-06-03T04:45:28.019 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: echo '0.000000 == .800000' 2024-06-03T04:45:28.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: (( 0 )) 2024-06-03T04:45:28.022 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: ceph config show osd.0 2024-06-03T04:45:28.022 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: grep osd_mclock_scheduler_background_recovery_res 2024-06-03T04:45:28.022 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: awk '{ print $2 }' 2024-06-03T04:45:28.022 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: bc 2024-06-03T04:45:28.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: res=.700000 2024-06-03T04:45:28.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:354: TEST_profile_disallow_builtin_params_modify: echo 'Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000' 2024-06-03T04:45:28.446 INFO:tasks.workunit.client.0.smithi088.stdout:Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000 2024-06-03T04:45:28.446 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.700000 == .800000' 2024-06-03T04:45:28.446 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-06-03T04:45:28.448 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.700000 != 0.700000' 2024-06-03T04:45:28.448 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-06-03T04:45:28.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-06-03T04:45:28.451 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_background_recovery_res 2024-06-03T04:45:28.451 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: bc 2024-06-03T04:45:28.451 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-06-03T04:45:28.451 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:45:28.451 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:45:28.451 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:45:28.451 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:45:28.451 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:45:28.452 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-osd.0.asok 2024-06-03T04:45:28.452 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-06-03T04:45:28.452 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.20650/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-06-03T04:45:28.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: res=0.700000 2024-06-03T04:45:28.556 INFO:tasks.workunit.client.0.smithi088.stdout:Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000 2024-06-03T04:45:28.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:363: TEST_profile_disallow_builtin_params_modify: echo 'Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000' 2024-06-03T04:45:28.557 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.700000 == .800000' 2024-06-03T04:45:28.557 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-06-03T04:45:28.558 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.700000 != 0.700000' 2024-06-03T04:45:28.558 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-06-03T04:45:28.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-06-03T04:45:28.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:370: TEST_profile_disallow_builtin_params_modify: '[' 0 -eq 0 ']' 2024-06-03T04:45:28.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:372: TEST_profile_disallow_builtin_params_modify: break 2024-06-03T04:45:28.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:329: TEST_profile_disallow_builtin_params_modify: for opt in "${options[@]}" 2024-06-03T04:45:28.561 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_client_res 2024-06-03T04:45:28.561 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: bc 2024-06-03T04:45:28.561 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-06-03T04:45:28.561 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:45:28.561 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:45:28.562 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:45:28.562 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:45:28.563 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:45:28.563 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-osd.0.asok 2024-06-03T04:45:28.563 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-06-03T04:45:28.563 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.20650/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-06-03T04:45:28.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: local opt_val_orig=0.300000 2024-06-03T04:45:28.667 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: bc -l 2024-06-03T04:45:28.667 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: echo '0.300000 + 0.1' 2024-06-03T04:45:28.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: local opt_val_new=.400000 2024-06-03T04:45:28.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:335: TEST_profile_disallow_builtin_params_modify: ceph config set osd.0 osd_mclock_scheduler_client_res .400000 2024-06-03T04:45:29.297 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: expr 10 - 1 2024-06-03T04:45:29.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: seq 0 9 2024-06-03T04:45:29.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: for count in $(seq 0 $(expr $retries - 1)) 2024-06-03T04:45:29.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:340: TEST_profile_disallow_builtin_params_modify: errors=0 2024-06-03T04:45:29.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:341: TEST_profile_disallow_builtin_params_modify: sleep 2 2024-06-03T04:45:31.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:343: TEST_profile_disallow_builtin_params_modify: echo 'Check configuration values - Attempt#: 0' 2024-06-03T04:45:31.301 INFO:tasks.workunit.client.0.smithi088.stdout:Check configuration values - Attempt#: 0 2024-06-03T04:45:31.301 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-06-03T04:45:31.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: local res=0.000000 2024-06-03T04:45:31.720 INFO:tasks.workunit.client.0.smithi088.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000 2024-06-03T04:45:31.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:346: TEST_profile_disallow_builtin_params_modify: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000' 2024-06-03T04:45:31.721 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: bc -l 2024-06-03T04:45:31.721 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: echo '0.000000 == .400000' 2024-06-03T04:45:31.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: (( 0 )) 2024-06-03T04:45:31.723 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: ceph config show osd.0 2024-06-03T04:45:31.723 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: grep osd_mclock_scheduler_client_res 2024-06-03T04:45:31.723 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: awk '{ print $2 }' 2024-06-03T04:45:31.723 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: bc 2024-06-03T04:45:32.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: res=.400000 2024-06-03T04:45:32.157 INFO:tasks.workunit.client.0.smithi088.stdout:Running config: osd.0 osd_mclock_scheduler_client_res = .400000 2024-06-03T04:45:32.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:354: TEST_profile_disallow_builtin_params_modify: echo 'Running config: osd.0 osd_mclock_scheduler_client_res = .400000' 2024-06-03T04:45:32.158 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.400000 == .400000' 2024-06-03T04:45:32.158 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-06-03T04:45:32.161 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.400000 != 0.300000' 2024-06-03T04:45:32.161 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-06-03T04:45:32.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: (( 1 || 1 )) 2024-06-03T04:45:32.163 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:357: TEST_profile_disallow_builtin_params_modify: expr 0 + 1 2024-06-03T04:45:32.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:357: TEST_profile_disallow_builtin_params_modify: errors=1 2024-06-03T04:45:32.165 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_client_res 2024-06-03T04:45:32.165 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: bc 2024-06-03T04:45:32.165 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-06-03T04:45:32.165 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:45:32.165 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:45:32.166 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:45:32.166 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:45:32.166 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:45:32.166 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-osd.0.asok 2024-06-03T04:45:32.166 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-06-03T04:45:32.166 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.20650/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-06-03T04:45:32.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: res=0.300000 2024-06-03T04:45:32.275 INFO:tasks.workunit.client.0.smithi088.stdout:Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000 2024-06-03T04:45:32.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:363: TEST_profile_disallow_builtin_params_modify: echo 'Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000' 2024-06-03T04:45:32.276 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-06-03T04:45:32.276 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.300000 == .400000' 2024-06-03T04:45:32.277 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.300000 != 0.300000' 2024-06-03T04:45:32.278 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-06-03T04:45:32.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-06-03T04:45:32.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:370: TEST_profile_disallow_builtin_params_modify: '[' 1 -eq 0 ']' 2024-06-03T04:45:32.279 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:373: TEST_profile_disallow_builtin_params_modify: expr 10 - 1 2024-06-03T04:45:32.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:373: TEST_profile_disallow_builtin_params_modify: '[' 0 -eq 9 ']' 2024-06-03T04:45:32.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: for count in $(seq 0 $(expr $retries - 1)) 2024-06-03T04:45:32.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:340: TEST_profile_disallow_builtin_params_modify: errors=0 2024-06-03T04:45:32.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:341: TEST_profile_disallow_builtin_params_modify: sleep 2 2024-06-03T04:45:34.283 INFO:tasks.workunit.client.0.smithi088.stdout:Check configuration values - Attempt#: 1 2024-06-03T04:45:34.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:343: TEST_profile_disallow_builtin_params_modify: echo 'Check configuration values - Attempt#: 1' 2024-06-03T04:45:34.284 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-06-03T04:45:34.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: local res=0.000000 2024-06-03T04:45:34.786 INFO:tasks.workunit.client.0.smithi088.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000 2024-06-03T04:45:34.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:346: TEST_profile_disallow_builtin_params_modify: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000' 2024-06-03T04:45:34.788 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: echo '0.000000 == .400000' 2024-06-03T04:45:34.788 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: bc -l 2024-06-03T04:45:34.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: (( 0 )) 2024-06-03T04:45:34.788 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: ceph config show osd.0 2024-06-03T04:45:34.788 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: grep osd_mclock_scheduler_client_res 2024-06-03T04:45:34.788 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: bc 2024-06-03T04:45:34.788 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: awk '{ print $2 }' 2024-06-03T04:45:35.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: res=.300000 2024-06-03T04:45:35.101 INFO:tasks.workunit.client.0.smithi088.stdout:Running config: osd.0 osd_mclock_scheduler_client_res = .300000 2024-06-03T04:45:35.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:354: TEST_profile_disallow_builtin_params_modify: echo 'Running config: osd.0 osd_mclock_scheduler_client_res = .300000' 2024-06-03T04:45:35.102 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.300000 == .400000' 2024-06-03T04:45:35.103 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-06-03T04:45:35.104 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.300000 != 0.300000' 2024-06-03T04:45:35.104 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-06-03T04:45:35.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-06-03T04:45:35.108 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_client_res 2024-06-03T04:45:35.108 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: bc 2024-06-03T04:45:35.108 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-06-03T04:45:35.108 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:45:35.109 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:45:35.109 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:45:35.109 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:45:35.109 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:45:35.109 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-osd.0.asok 2024-06-03T04:45:35.109 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-06-03T04:45:35.109 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.20650/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-06-03T04:45:35.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: res=0.300000 2024-06-03T04:45:35.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:363: TEST_profile_disallow_builtin_params_modify: echo 'Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000' 2024-06-03T04:45:35.213 INFO:tasks.workunit.client.0.smithi088.stdout:Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000 2024-06-03T04:45:35.213 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.300000 == .400000' 2024-06-03T04:45:35.213 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-06-03T04:45:35.215 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.300000 != 0.300000' 2024-06-03T04:45:35.215 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-06-03T04:45:35.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-06-03T04:45:35.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:370: TEST_profile_disallow_builtin_params_modify: '[' 0 -eq 0 ']' 2024-06-03T04:45:35.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:372: TEST_profile_disallow_builtin_params_modify: break 2024-06-03T04:45:35.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:380: TEST_profile_disallow_builtin_params_modify: teardown td/mclock-config 2024-06-03T04:45:35.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-03T04:45:35.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T04:45:35.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-03T04:45:35.217 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:45:35.217 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:45:35.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:45:35.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:45:35.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:45:35.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:45:35.323 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:45:35.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:45:35.324 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:45:35.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:45:35.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:45:35.325 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:45:35.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:45:35.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:45:35.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:45:35.327 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:45:35.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:45:35.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T04:45:35.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-03T04:45:35.333 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:45:35.334 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:45:35.334 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:45:35.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20650 2024-06-03T04:45:35.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:45:35.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:45:35.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-06-03T04:45:35.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-03T04:45:35.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T04:45:35.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-03T04:45:35.335 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:45:35.336 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:45:35.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:45:35.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:45:35.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:45:35.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:45:35.337 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:45:35.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:45:35.338 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:45:35.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:45:35.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:45:35.339 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:45:35.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:45:35.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:45:35.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:45:35.340 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:45:35.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:45:35.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T04:45:35.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-03T04:45:35.343 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:45:35.343 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:45:35.343 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:45:35.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20650 2024-06-03T04:45:35.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:45:35.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:45:35.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-06-03T04:45:35.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-06-03T04:45:35.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-06-03T04:45:35.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-06-03T04:45:35.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-03T04:45:35.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T04:45:35.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-03T04:45:35.345 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:45:35.346 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:45:35.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:45:35.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:45:35.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:45:35.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:45:35.347 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:45:35.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:45:35.347 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:45:35.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:45:35.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:45:35.348 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:45:35.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:45:35.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:45:35.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:45:35.349 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:45:35.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:45:35.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T04:45:35.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-03T04:45:35.351 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:45:35.351 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:45:35.351 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:45:35.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20650 2024-06-03T04:45:35.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:45:35.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:45:35.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-06-03T04:45:35.353 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T04:45:35.353 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:45:35.353 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:45:35.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20650 2024-06-03T04:45:35.356 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T04:45:35.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T04:45:35.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T04:45:35.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-06-03T04:45:35.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_disallow_builtin_params_override td/mclock-config 2024-06-03T04:45:35.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:384: TEST_profile_disallow_builtin_params_override: local dir=td/mclock-config 2024-06-03T04:45:35.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:386: TEST_profile_disallow_builtin_params_override: setup td/mclock-config 2024-06-03T04:45:35.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-06-03T04:45:35.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-06-03T04:45:35.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-03T04:45:35.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T04:45:35.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-03T04:45:35.356 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:45:35.357 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:45:35.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:45:35.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:45:35.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:45:35.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:45:35.357 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:45:35.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:45:35.358 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:45:35.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:45:35.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:45:35.359 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:45:35.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:45:35.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:45:35.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:45:35.360 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:45:35.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:45:35.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T04:45:35.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-03T04:45:35.363 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:45:35.363 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:45:35.363 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:45:35.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20650 2024-06-03T04:45:35.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:45:35.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:45:35.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-06-03T04:45:35.365 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T04:45:35.365 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:45:35.365 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:45:35.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20650 2024-06-03T04:45:35.367 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T04:45:35.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T04:45:35.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T04:45:35.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-06-03T04:45:35.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:387: TEST_profile_disallow_builtin_params_override: run_mon td/mclock-config a 2024-06-03T04:45:35.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-06-03T04:45:35.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T04:45:35.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T04:45:35.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T04:45:35.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-06-03T04:45:35.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-06-03T04:45:35.790 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T04:45:35.790 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:45:35.790 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:45:35.791 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:45:35.791 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:45:35.791 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:45:35.792 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20650/$cluster-$name.asok' 2024-06-03T04:45:35.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.20650/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-03T04:45:36.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T04:45:36.028 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T04:45:36.028 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:45:36.028 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T04:45:36.028 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T04:45:36.029 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T04:45:36.029 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T04:45:36.029 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T04:45:36.029 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T04:45:36.030 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:45:36.030 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:45:36.030 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:45:36.030 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-mon.a.asok 2024-06-03T04:45:36.031 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:45:36.031 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20650/ceph-mon.a.asok config get fsid 2024-06-03T04:45:36.139 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T04:45:36.139 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:45:36.139 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T04:45:36.139 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T04:45:36.139 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T04:45:36.140 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T04:45:36.140 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T04:45:36.141 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T04:45:36.141 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:45:36.141 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:45:36.141 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:45:36.141 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-mon.a.asok 2024-06-03T04:45:36.141 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:45:36.141 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20650/ceph-mon.a.asok config get mon_host 2024-06-03T04:45:36.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:388: TEST_profile_disallow_builtin_params_override: run_mgr td/mclock-config x 2024-06-03T04:45:36.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-06-03T04:45:36.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T04:45:36.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T04:45:36.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T04:45:36.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-06-03T04:45:36.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T04:45:37.150 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T04:45:37.150 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:45:37.150 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:45:37.150 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:45:37.150 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:45:37.150 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:45:37.150 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20650/$cluster-$name.asok' 2024-06-03T04:45:37.151 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T04:45:37.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.20650/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T04:45:37.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:390: TEST_profile_disallow_builtin_params_override: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-06-03T04:45:37.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-06-03T04:45:37.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T04:45:37.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T04:45:37.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T04:45:37.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-06-03T04:45:37.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=78136f38-0d6f-4bc0-8fbc-da8924e531b4 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-06-03T04:45:37.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T04:45:37.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T04:45:37.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T04:45:37.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-06-03T04:45:37.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-06-03T04:45:37.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T04:45:37.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T04:45:37.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-06-03T04:45:37.193 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T04:45:37.193 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:45:37.193 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:45:37.193 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:45:37.193 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:45:37.193 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:45:37.193 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20650/$cluster-$name.asok' 2024-06-03T04:45:37.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20650/$cluster-$name.asok' 2024-06-03T04:45:37.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T04:45:37.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T04:45:37.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T04:45:37.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-06-03T04:45:37.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-06-03T04:45:37.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T04:45:37.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T04:45:37.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T04:45:37.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T04:45:37.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T04:45:37.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-06-03T04:45:37.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-06-03T04:45:37.195 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T04:45:37.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8a7d78cc-1455-43b3-8247-195f8cd220d6 2024-06-03T04:45:37.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8a7d78cc-1455-43b3-8247-195f8cd220d6' 2024-06-03T04:45:37.195 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 8a7d78cc-1455-43b3-8247-195f8cd220d6 2024-06-03T04:45:37.195 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T04:45:37.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBxSl1mgZV7DBAAfBEhYRVbe5DGxHmmKUF5rA== 2024-06-03T04:45:37.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBxSl1mgZV7DBAAfBEhYRVbe5DGxHmmKUF5rA=="}' 2024-06-03T04:45:37.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8a7d78cc-1455-43b3-8247-195f8cd220d6 -i td/mclock-config/0/new.json 2024-06-03T04:45:37.523 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T04:45:37.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-06-03T04:45:37.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=78136f38-0d6f-4bc0-8fbc-da8924e531b4 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.20650/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQBxSl1mgZV7DBAAfBEhYRVbe5DGxHmmKUF5rA== --osd-uuid 8a7d78cc-1455-43b3-8247-195f8cd220d6 2024-06-03T04:45:37.577 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:45:37.574+0000 7f72714de7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:45:37.591 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:45:37.586+0000 7f72714de7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:45:37.592 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:45:37.586+0000 7f72714de7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:45:37.593 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:45:37.586+0000 7f72714de7c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-06-03T04:45:37.593 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:45:37.586+0000 7f72714de7c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-06-03T04:45:37.593 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:45:37.586+0000 7f72714de7c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-06-03T04:45:37.640 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:45:37.634+0000 7f72714de7c0 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-06-03T04:45:44.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-06-03T04:45:44.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T04:45:44.004 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T04:45:44.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T04:45:44.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T04:45:44.540 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T04:45:44.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T04:45:44.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T04:45:44.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T04:45:44.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T04:45:44.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=78136f38-0d6f-4bc0-8fbc-da8924e531b4 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.20650/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-06-03T04:45:44.572 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:45:44.566+0000 7f8f72d777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:45:44.575 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:45:44.570+0000 7f8f72d777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:45:44.576 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:45:44.570+0000 7f8f72d777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:45:44.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T04:45:44.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T04:45:44.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T04:45:44.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T04:45:44.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T04:45:44.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:45:44.950 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T04:45:44.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T04:45:44.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:45:44.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:45:45.127 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:45:45.122+0000 7f8f72d777c0 -1 Falling back to public interface 2024-06-03T04:45:45.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:45:46.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:45:46.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:45:46.354 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T04:45:46.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T04:45:46.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:45:46.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:45:46.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:45:47.673 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:45:47.666+0000 7f8f72d777c0 -1 osd.0 0 log_to_monitors true 2024-06-03T04:45:47.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:45:47.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:45:47.759 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T04:45:47.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T04:45:47.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:45:47.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:45:48.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:45:49.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:45:49.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:45:49.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T04:45:49.396 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T04:45:49.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:45:49.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:45:49.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:45:50.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:45:50.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:45:50.829 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T04:45:50.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T04:45:50.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:45:50.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:45:51.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:45:52.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:45:52.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:45:52.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T04:45:52.243 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T04:45:52.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:45:52.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:45:52.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:45:53.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:45:53.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:45:53.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T04:45:53.697 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T04:45:53.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:45:53.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:45:54.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:45:55.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:45:55.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:45:55.206 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T04:45:55.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T04:45:55.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:45:55.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:45:55.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:45:56.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:45:56.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:45:56.642 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T04:45:56.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T04:45:56.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:45:56.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:45:57.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:45:58.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:45:58.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:45:58.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T04:45:58.103 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T04:45:58.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:45:58.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:45:58.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:45:59.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:45:59.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:45:59.672 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T04:45:59.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T04:45:59.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:45:59.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:46:00.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:46:01.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:46:01.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:46:01.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T04:46:01.129 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T04:46:01.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:46:01.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:46:02.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:46:03.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:46:03.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:46:03.021 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T04:46:03.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T04:46:03.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:46:03.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:46:03.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:46:04.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:46:04.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:46:04.911 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T04:46:04.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T04:46:04.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:46:04.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:46:05.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:46:06.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:46:06.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:46:06.669 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T04:46:06.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T04:46:06.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:46:06.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:46:07.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:46:08.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:46:08.203 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T04:46:08.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:46:08.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T04:46:08.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:46:08.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:46:09.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:46:10.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:46:10.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:46:10.279 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T04:46:10.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T04:46:10.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:46:10.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:46:11.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:46:12.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:46:12.107 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T04:46:12.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:46:12.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T04:46:12.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:46:12.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:46:12.559 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2417734162,v1:127.0.0.1:6803/2417734162] [v2:127.0.0.1:6804/2417734162,v1:127.0.0.1:6805/2417734162] exists,up 8a7d78cc-1455-43b3-8247-195f8cd220d6 2024-06-03T04:46:12.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T04:46:12.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T04:46:12.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T04:46:12.560 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:393: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_profile 2024-06-03T04:46:12.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:393: TEST_profile_disallow_builtin_params_override: local def_mclock_profile=balanced 2024-06-03T04:46:12.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:394: TEST_profile_disallow_builtin_params_override: test balanced = balanced 2024-06-03T04:46:12.969 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_profile 2024-06-03T04:46:12.969 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-06-03T04:46:12.969 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:46:12.969 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:46:12.970 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:46:12.970 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:46:12.970 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:46:12.970 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-osd.0.asok 2024-06-03T04:46:12.970 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-06-03T04:46:12.971 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.20650/ceph-osd.0.asok config get osd_mclock_profile 2024-06-03T04:46:13.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: local 'cur_mclock_profile="high_recovery_ops"' 2024-06-03T04:46:13.090 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: eval echo '"high_recovery_ops"' 2024-06-03T04:46:13.090 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: echo high_recovery_ops 2024-06-03T04:46:13.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: cur_mclock_profile=high_recovery_ops 2024-06-03T04:46:13.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:401: TEST_profile_disallow_builtin_params_override: test high_recovery_ops = high_recovery_ops 2024-06-03T04:46:13.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:403: TEST_profile_disallow_builtin_params_override: options=('osd_mclock_scheduler_background_recovery_res' 'osd_mclock_scheduler_client_res') 2024-06-03T04:46:13.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:403: TEST_profile_disallow_builtin_params_override: declare -a options 2024-06-03T04:46:13.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:406: TEST_profile_disallow_builtin_params_override: local retries=10 2024-06-03T04:46:13.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:407: TEST_profile_disallow_builtin_params_override: local errors=0 2024-06-03T04:46:13.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:408: TEST_profile_disallow_builtin_params_override: for opt in "${options[@]}" 2024-06-03T04:46:13.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: bc 2024-06-03T04:46:13.093 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_background_recovery_res 2024-06-03T04:46:13.093 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-06-03T04:46:13.093 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:46:13.093 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:46:13.093 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:46:13.093 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:46:13.093 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:46:13.093 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-osd.0.asok 2024-06-03T04:46:13.094 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-06-03T04:46:13.094 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.20650/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-06-03T04:46:13.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: local opt_val_orig=0.700000 2024-06-03T04:46:13.200 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: echo '0.700000 + 0.1' 2024-06-03T04:46:13.200 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: bc -l 2024-06-03T04:46:13.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: local opt_val_new=.800000 2024-06-03T04:46:13.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:414: TEST_profile_disallow_builtin_params_override: ceph tell osd.0 config set osd_mclock_scheduler_background_recovery_res .800000 2024-06-03T04:46:13.339 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-06-03T04:46:13.339 INFO:tasks.workunit.client.0.smithi088.stdout: "success": "osd_mclock_scheduler_background_recovery_res = '' " 2024-06-03T04:46:13.339 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-06-03T04:46:13.353 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: expr 10 - 1 2024-06-03T04:46:13.354 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: seq 0 9 2024-06-03T04:46:13.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: for count in $(seq 0 $(expr $retries - 1)) 2024-06-03T04:46:13.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:419: TEST_profile_disallow_builtin_params_override: errors=0 2024-06-03T04:46:13.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:420: TEST_profile_disallow_builtin_params_override: sleep 2 2024-06-03T04:46:15.356 INFO:tasks.workunit.client.0.smithi088.stdout:Check configuration values - Attempt#: 0 2024-06-03T04:46:15.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:422: TEST_profile_disallow_builtin_params_override: echo 'Check configuration values - Attempt#: 0' 2024-06-03T04:46:15.357 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_scheduler_background_recovery_res 2024-06-03T04:46:15.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: local res=0.000000 2024-06-03T04:46:15.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:425: TEST_profile_disallow_builtin_params_override: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000' 2024-06-03T04:46:15.775 INFO:tasks.workunit.client.0.smithi088.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000 2024-06-03T04:46:15.776 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: echo '0.000000 == .800000' 2024-06-03T04:46:15.776 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: bc -l 2024-06-03T04:46:15.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: (( 0 )) 2024-06-03T04:46:15.779 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: ceph config show osd.0 2024-06-03T04:46:15.779 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: grep osd_mclock_scheduler_background_recovery_res 2024-06-03T04:46:15.779 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: awk '{ print $2 }' 2024-06-03T04:46:15.779 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: bc 2024-06-03T04:46:16.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: res=.700000 2024-06-03T04:46:16.203 INFO:tasks.workunit.client.0.smithi088.stdout:Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000 2024-06-03T04:46:16.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:433: TEST_profile_disallow_builtin_params_override: echo 'Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000' 2024-06-03T04:46:16.204 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.700000 == .800000' 2024-06-03T04:46:16.204 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-06-03T04:46:16.206 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.700000 != 0.700000' 2024-06-03T04:46:16.206 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-06-03T04:46:16.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-06-03T04:46:16.209 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_background_recovery_res 2024-06-03T04:46:16.209 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: bc 2024-06-03T04:46:16.209 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-06-03T04:46:16.209 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:46:16.209 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:46:16.209 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:46:16.209 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:46:16.209 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:46:16.209 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-osd.0.asok 2024-06-03T04:46:16.210 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-06-03T04:46:16.210 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.20650/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-06-03T04:46:16.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: res=0.700000 2024-06-03T04:46:16.322 INFO:tasks.workunit.client.0.smithi088.stdout:Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000 2024-06-03T04:46:16.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:442: TEST_profile_disallow_builtin_params_override: echo 'Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000' 2024-06-03T04:46:16.323 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.700000 == .800000' 2024-06-03T04:46:16.323 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-06-03T04:46:16.326 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.700000 != 0.700000' 2024-06-03T04:46:16.326 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-06-03T04:46:16.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-06-03T04:46:16.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:449: TEST_profile_disallow_builtin_params_override: '[' 0 -eq 0 ']' 2024-06-03T04:46:16.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:451: TEST_profile_disallow_builtin_params_override: break 2024-06-03T04:46:16.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:408: TEST_profile_disallow_builtin_params_override: for opt in "${options[@]}" 2024-06-03T04:46:16.328 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-06-03T04:46:16.328 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:46:16.328 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: bc 2024-06-03T04:46:16.328 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:46:16.328 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_client_res 2024-06-03T04:46:16.329 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:46:16.329 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:46:16.329 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:46:16.329 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-osd.0.asok 2024-06-03T04:46:16.329 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-06-03T04:46:16.330 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.20650/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-06-03T04:46:16.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: local opt_val_orig=0.300000 2024-06-03T04:46:16.445 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: echo '0.300000 + 0.1' 2024-06-03T04:46:16.445 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: bc -l 2024-06-03T04:46:16.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: local opt_val_new=.400000 2024-06-03T04:46:16.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:414: TEST_profile_disallow_builtin_params_override: ceph tell osd.0 config set osd_mclock_scheduler_client_res .400000 2024-06-03T04:46:16.594 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-06-03T04:46:16.594 INFO:tasks.workunit.client.0.smithi088.stdout: "success": "osd_mclock_scheduler_client_res = '' " 2024-06-03T04:46:16.595 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-06-03T04:46:16.609 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: expr 10 - 1 2024-06-03T04:46:16.610 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: seq 0 9 2024-06-03T04:46:16.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: for count in $(seq 0 $(expr $retries - 1)) 2024-06-03T04:46:16.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:419: TEST_profile_disallow_builtin_params_override: errors=0 2024-06-03T04:46:16.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:420: TEST_profile_disallow_builtin_params_override: sleep 2 2024-06-03T04:46:18.612 INFO:tasks.workunit.client.0.smithi088.stdout:Check configuration values - Attempt#: 0 2024-06-03T04:46:18.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:422: TEST_profile_disallow_builtin_params_override: echo 'Check configuration values - Attempt#: 0' 2024-06-03T04:46:18.613 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-06-03T04:46:19.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: local res=0.000000 2024-06-03T04:46:19.010 INFO:tasks.workunit.client.0.smithi088.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000 2024-06-03T04:46:19.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:425: TEST_profile_disallow_builtin_params_override: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000' 2024-06-03T04:46:19.011 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: echo '0.000000 == .400000' 2024-06-03T04:46:19.011 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: bc -l 2024-06-03T04:46:19.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: (( 0 )) 2024-06-03T04:46:19.014 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: ceph config show osd.0 2024-06-03T04:46:19.015 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: grep osd_mclock_scheduler_client_res 2024-06-03T04:46:19.015 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: awk '{ print $2 }' 2024-06-03T04:46:19.015 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: bc 2024-06-03T04:46:19.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: res=.300000 2024-06-03T04:46:19.413 INFO:tasks.workunit.client.0.smithi088.stdout:Running config: osd.0 osd_mclock_scheduler_client_res = .300000 2024-06-03T04:46:19.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:433: TEST_profile_disallow_builtin_params_override: echo 'Running config: osd.0 osd_mclock_scheduler_client_res = .300000' 2024-06-03T04:46:19.413 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.300000 == .400000' 2024-06-03T04:46:19.413 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-06-03T04:46:19.415 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.300000 != 0.300000' 2024-06-03T04:46:19.415 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-06-03T04:46:19.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-06-03T04:46:19.418 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_client_res 2024-06-03T04:46:19.418 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: bc 2024-06-03T04:46:19.418 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-06-03T04:46:19.418 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:46:19.418 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:46:19.419 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:46:19.419 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:46:19.419 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:46:19.419 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-osd.0.asok 2024-06-03T04:46:19.419 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-06-03T04:46:19.419 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.20650/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-06-03T04:46:19.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: res=0.300000 2024-06-03T04:46:19.524 INFO:tasks.workunit.client.0.smithi088.stdout:Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000 2024-06-03T04:46:19.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:442: TEST_profile_disallow_builtin_params_override: echo 'Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000' 2024-06-03T04:46:19.525 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-06-03T04:46:19.525 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.300000 == .400000' 2024-06-03T04:46:19.527 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-06-03T04:46:19.527 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.300000 != 0.300000' 2024-06-03T04:46:19.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-06-03T04:46:19.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:449: TEST_profile_disallow_builtin_params_override: '[' 0 -eq 0 ']' 2024-06-03T04:46:19.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:451: TEST_profile_disallow_builtin_params_override: break 2024-06-03T04:46:19.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:459: TEST_profile_disallow_builtin_params_override: teardown td/mclock-config 2024-06-03T04:46:19.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-03T04:46:19.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T04:46:19.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-03T04:46:19.530 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:46:19.530 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:46:19.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:46:19.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:46:19.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:46:19.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:46:19.838 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:46:19.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:46:19.839 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:46:19.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:46:19.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:46:19.840 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:46:19.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:46:19.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:46:19.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:46:19.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:46:19.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:46:19.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T04:46:19.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-03T04:46:19.857 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:46:19.857 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:46:19.857 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:46:19.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20650 2024-06-03T04:46:19.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:46:19.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:46:19.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-06-03T04:46:19.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-03T04:46:19.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T04:46:19.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-03T04:46:19.858 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:46:19.859 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:46:19.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:46:19.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:46:19.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:46:19.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:46:19.862 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:46:19.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:46:19.862 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:46:19.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:46:19.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:46:19.863 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:46:19.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:46:19.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:46:19.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:46:19.863 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:46:19.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:46:19.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T04:46:19.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-03T04:46:19.868 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:46:19.868 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:46:19.868 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:46:19.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20650 2024-06-03T04:46:19.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:46:19.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:46:19.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-06-03T04:46:19.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-06-03T04:46:19.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-06-03T04:46:19.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-06-03T04:46:19.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-03T04:46:19.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T04:46:19.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-03T04:46:19.870 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:46:19.870 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:46:19.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:46:19.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:46:19.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:46:19.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:46:19.873 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:46:19.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:46:19.874 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:46:19.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:46:19.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:46:19.874 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:46:19.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:46:19.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:46:19.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:46:19.875 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:46:19.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:46:19.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T04:46:19.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-03T04:46:19.878 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:46:19.878 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:46:19.878 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:46:19.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20650 2024-06-03T04:46:19.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:46:19.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:46:19.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-06-03T04:46:19.884 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T04:46:19.884 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:46:19.884 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:46:19.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20650 2024-06-03T04:46:19.885 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T04:46:19.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T04:46:19.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T04:46:19.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-06-03T04:46:19.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_recovery_limit_adjustment_mclock td/mclock-config 2024-06-03T04:46:19.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:202: TEST_recovery_limit_adjustment_mclock: local dir=td/mclock-config 2024-06-03T04:46:19.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:204: TEST_recovery_limit_adjustment_mclock: setup td/mclock-config 2024-06-03T04:46:19.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-06-03T04:46:19.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-06-03T04:46:19.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-03T04:46:19.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T04:46:19.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-03T04:46:19.887 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:46:19.887 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:46:19.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:46:19.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:46:19.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:46:19.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:46:19.888 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:46:19.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:46:19.888 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:46:19.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:46:19.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:46:19.892 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:46:19.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:46:19.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:46:19.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:46:19.892 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:46:19.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:46:19.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T04:46:19.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-03T04:46:19.893 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:46:19.893 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:46:19.893 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:46:19.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20650 2024-06-03T04:46:19.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:46:19.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:46:19.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-06-03T04:46:19.897 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T04:46:19.898 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:46:19.898 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:46:19.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20650 2024-06-03T04:46:19.898 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T04:46:19.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T04:46:19.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T04:46:19.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-06-03T04:46:19.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:205: TEST_recovery_limit_adjustment_mclock: run_mon td/mclock-config a 2024-06-03T04:46:19.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-06-03T04:46:19.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T04:46:19.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T04:46:19.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T04:46:19.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-06-03T04:46:19.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-06-03T04:46:20.305 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T04:46:20.305 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:46:20.305 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:46:20.306 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:46:20.306 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:46:20.306 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:46:20.306 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20650/$cluster-$name.asok' 2024-06-03T04:46:20.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.20650/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-03T04:46:20.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T04:46:20.524 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T04:46:20.524 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:46:20.524 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T04:46:20.524 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T04:46:20.525 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T04:46:20.525 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T04:46:20.525 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T04:46:20.525 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T04:46:20.525 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:46:20.525 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:46:20.525 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:46:20.526 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-mon.a.asok 2024-06-03T04:46:20.526 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:46:20.526 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20650/ceph-mon.a.asok config get fsid 2024-06-03T04:46:20.639 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T04:46:20.639 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:46:20.639 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T04:46:20.639 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T04:46:20.640 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T04:46:20.640 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T04:46:20.640 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T04:46:20.640 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T04:46:20.641 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:46:20.641 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:46:20.641 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:46:20.641 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-mon.a.asok 2024-06-03T04:46:20.642 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:46:20.642 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20650/ceph-mon.a.asok config get mon_host 2024-06-03T04:46:20.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:206: TEST_recovery_limit_adjustment_mclock: run_mgr td/mclock-config x 2024-06-03T04:46:20.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-06-03T04:46:20.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T04:46:20.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T04:46:20.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T04:46:20.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-06-03T04:46:20.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T04:46:21.644 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T04:46:21.644 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:46:21.644 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:46:21.644 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:46:21.645 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:46:21.645 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:46:21.645 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20650/$cluster-$name.asok' 2024-06-03T04:46:21.645 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T04:46:21.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.20650/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T04:46:21.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:208: TEST_recovery_limit_adjustment_mclock: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-06-03T04:46:21.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-06-03T04:46:21.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T04:46:21.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T04:46:21.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T04:46:21.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-06-03T04:46:21.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=78136f38-0d6f-4bc0-8fbc-da8924e531b4 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-06-03T04:46:21.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T04:46:21.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T04:46:21.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T04:46:21.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-06-03T04:46:21.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-06-03T04:46:21.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T04:46:21.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T04:46:21.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-06-03T04:46:21.683 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T04:46:21.683 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:46:21.683 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:46:21.683 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:46:21.683 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:46:21.683 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:46:21.683 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20650/$cluster-$name.asok' 2024-06-03T04:46:21.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20650/$cluster-$name.asok' 2024-06-03T04:46:21.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T04:46:21.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T04:46:21.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T04:46:21.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-06-03T04:46:21.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-06-03T04:46:21.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T04:46:21.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T04:46:21.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T04:46:21.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T04:46:21.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T04:46:21.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-06-03T04:46:21.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-06-03T04:46:21.685 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T04:46:21.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cef5c831-09cc-4b3f-9092-a163181269d0 2024-06-03T04:46:21.685 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 cef5c831-09cc-4b3f-9092-a163181269d0 2024-06-03T04:46:21.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 cef5c831-09cc-4b3f-9092-a163181269d0' 2024-06-03T04:46:21.686 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T04:46:21.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCdSl1mT6KcKRAAYEKAXWa7L0uahQaDhsak7Q== 2024-06-03T04:46:21.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCdSl1mT6KcKRAAYEKAXWa7L0uahQaDhsak7Q=="}' 2024-06-03T04:46:21.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cef5c831-09cc-4b3f-9092-a163181269d0 -i td/mclock-config/0/new.json 2024-06-03T04:46:22.004 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T04:46:22.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-06-03T04:46:22.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=78136f38-0d6f-4bc0-8fbc-da8924e531b4 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.20650/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQCdSl1mT6KcKRAAYEKAXWa7L0uahQaDhsak7Q== --osd-uuid cef5c831-09cc-4b3f-9092-a163181269d0 2024-06-03T04:46:22.054 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:46:22.049+0000 7fb8a95f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:46:22.090 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:46:22.085+0000 7fb8a95f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:46:22.092 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:46:22.085+0000 7fb8a95f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:46:22.092 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:46:22.085+0000 7fb8a95f37c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-06-03T04:46:22.092 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:46:22.085+0000 7fb8a95f37c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-06-03T04:46:22.092 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:46:22.085+0000 7fb8a95f37c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-06-03T04:46:22.138 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:46:22.133+0000 7fb8a95f37c0 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-06-03T04:46:28.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-06-03T04:46:28.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T04:46:28.407 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T04:46:28.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T04:46:28.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T04:46:28.949 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T04:46:28.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T04:46:28.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=78136f38-0d6f-4bc0-8fbc-da8924e531b4 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.20650/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-06-03T04:46:28.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T04:46:28.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T04:46:28.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T04:46:28.979 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:46:28.973+0000 7f9f9b3cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:46:29.024 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:46:29.017+0000 7f9f9b3cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:46:29.025 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:46:29.021+0000 7f9f9b3cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:46:29.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T04:46:29.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T04:46:29.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T04:46:29.391 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T04:46:29.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T04:46:29.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T04:46:29.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:46:29.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T04:46:29.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:46:29.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:46:29.587 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:46:29.581+0000 7f9f9b3cf7c0 -1 Falling back to public interface 2024-06-03T04:46:29.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:46:30.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:46:30.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:46:30.792 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T04:46:30.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T04:46:30.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:46:30.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:46:31.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:46:32.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:46:32.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:46:32.189 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T04:46:32.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T04:46:32.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:46:32.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:46:32.219 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:46:32.213+0000 7f9f9b3cf7c0 -1 osd.0 0 log_to_monitors true 2024-06-03T04:46:32.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:46:33.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:46:33.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:46:33.687 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T04:46:33.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T04:46:33.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:46:33.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:46:34.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:46:35.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:46:35.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:46:35.165 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T04:46:35.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T04:46:35.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:46:35.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:46:35.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:46:36.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:46:36.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:46:36.611 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T04:46:36.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T04:46:36.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:46:36.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:46:37.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:46:38.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:46:38.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:46:38.161 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T04:46:38.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T04:46:38.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:46:38.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:46:38.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:46:39.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:46:39.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:46:39.583 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T04:46:39.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T04:46:39.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:46:39.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:46:40.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:46:41.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:46:41.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:46:41.177 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T04:46:41.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T04:46:41.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:46:41.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:46:41.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:46:42.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:46:42.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:46:42.610 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T04:46:42.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T04:46:42.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:46:42.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:46:43.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:46:44.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:46:44.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:46:44.194 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T04:46:44.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T04:46:44.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:46:44.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:46:44.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:46:45.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:46:45.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:46:45.639 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T04:46:45.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T04:46:45.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:46:45.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:46:46.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:46:47.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:46:47.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:46:47.475 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T04:46:47.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T04:46:47.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:46:47.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:46:48.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:46:49.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:46:49.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:46:49.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T04:46:49.318 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T04:46:49.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:46:49.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:46:50.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:46:51.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:46:51.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:46:51.046 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T04:46:51.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T04:46:51.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:46:51.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:46:51.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:46:52.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:46:52.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:46:52.792 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T04:46:52.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T04:46:52.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:46:52.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:46:53.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:46:54.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:46:54.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:46:54.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T04:46:54.558 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T04:46:54.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:46:54.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:46:55.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:46:56.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:46:56.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:46:56.458 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T04:46:56.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T04:46:56.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:46:56.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:46:56.929 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2384973779,v1:127.0.0.1:6803/2384973779] [v2:127.0.0.1:6804/2384973779,v1:127.0.0.1:6805/2384973779] exists,up cef5c831-09cc-4b3f-9092-a163181269d0 2024-06-03T04:46:56.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T04:46:56.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T04:46:56.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T04:46:56.930 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2024-06-03T04:46:56.930 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:46:56.931 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:46:56.931 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:46:56.931 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:46:56.931 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:46:56.931 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-osd.0.asok 2024-06-03T04:46:56.932 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2024-06-03T04:46:56.932 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.20650/ceph-osd.0.asok config get osd_recovery_max_active 2024-06-03T04:46:57.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: local 'recoveries={"osd_recovery_max_active":"0"}' 2024-06-03T04:46:57.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:212: TEST_recovery_limit_adjustment_mclock: echo '{"osd_recovery_max_active":"0"}' 2024-06-03T04:46:57.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:212: TEST_recovery_limit_adjustment_mclock: grep --quiet osd_recovery_max_active 2024-06-03T04:46:57.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:217: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_recovery_max_active 10 2024-06-03T04:46:57.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:218: TEST_recovery_limit_adjustment_mclock: sleep 2 2024-06-03T04:46:59.876 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2024-06-03T04:46:59.876 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:46:59.876 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:46:59.876 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:46:59.876 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:46:59.876 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:46:59.877 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-osd.0.asok 2024-06-03T04:46:59.877 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2024-06-03T04:46:59.877 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.20650/ceph-osd.0.asok config get osd_recovery_max_active 2024-06-03T04:47:00.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: local 'max_recoveries={"osd_recovery_max_active":"0"}' 2024-06-03T04:47:00.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:221: TEST_recovery_limit_adjustment_mclock: test '{"osd_recovery_max_active":"0"}' = '{"osd_recovery_max_active":"0"}' 2024-06-03T04:47:00.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:225: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_mclock_override_recovery_settings true 2024-06-03T04:47:00.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:226: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_recovery_max_active 10 2024-06-03T04:47:01.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:227: TEST_recovery_limit_adjustment_mclock: sleep 2 2024-06-03T04:47:03.030 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2024-06-03T04:47:03.030 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:47:03.030 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:47:03.031 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:47:03.031 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:47:03.031 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:47:03.031 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20650/ceph-osd.0.asok 2024-06-03T04:47:03.031 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2024-06-03T04:47:03.031 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.20650/ceph-osd.0.asok config get osd_recovery_max_active 2024-06-03T04:47:03.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: max_recoveries='{"osd_recovery_max_active":"10"}' 2024-06-03T04:47:03.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:230: TEST_recovery_limit_adjustment_mclock: test '{"osd_recovery_max_active":"10"}' = '{"osd_recovery_max_active":"10"}' 2024-06-03T04:47:03.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:232: TEST_recovery_limit_adjustment_mclock: teardown td/mclock-config 2024-06-03T04:47:03.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-03T04:47:03.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T04:47:03.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-03T04:47:03.155 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:47:03.156 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:47:03.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:47:03.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:47:03.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:47:03.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:47:03.265 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:47:03.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:47:03.266 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:47:03.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:47:03.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:47:03.267 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:47:03.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:47:03.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:47:03.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:47:03.269 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:47:03.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:47:03.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T04:47:03.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-03T04:47:03.277 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:47:03.277 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:47:03.277 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:47:03.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20650 2024-06-03T04:47:03.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:47:03.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:47:03.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-06-03T04:47:03.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-03T04:47:03.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T04:47:03.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-03T04:47:03.280 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:47:03.280 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:47:03.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:47:03.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:47:03.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:47:03.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:47:03.281 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:47:03.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:47:03.282 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:47:03.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:47:03.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:47:03.283 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:47:03.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:47:03.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:47:03.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:47:03.284 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:47:03.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:47:03.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T04:47:03.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-03T04:47:03.289 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:47:03.290 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:47:03.290 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:47:03.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20650 2024-06-03T04:47:03.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:47:03.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:47:03.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-06-03T04:47:03.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/mclock-config 0 2024-06-03T04:47:03.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-03T04:47:03.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-06-03T04:47:03.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-03T04:47:03.291 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:47:03.291 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:47:03.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:47:03.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:47:03.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:47:03.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:47:03.292 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:47:03.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:47:03.293 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:47:03.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:47:03.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:47:03.294 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:47:03.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:47:03.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:47:03.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:47:03.295 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:47:03.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:47:03.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-06-03T04:47:03.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-03T04:47:03.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:47:03.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:47:03.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20650 2024-06-03T04:47:03.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20650 2024-06-03T04:47:03.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:47:03.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:47:03.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-06-03T04:47:03.301 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-06-03T04:47:03.301 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-06-03T04:47:03.351 INFO:tasks.workunit:Running workunit misc/network-ping.sh... 2024-06-03T04:47:03.351 DEBUG:teuthology.orchestra.run.smithi088:workunit test misc/network-ping.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dfcc4b3c2c55532c4c04fa47551c9df1dffdf746 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh 2024-06-03T04:47:03.404 INFO:tasks.workunit.client.0.smithi088.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-06-03T04:47:03.409 INFO:tasks.workunit.client.0.smithi088.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-06-03T04:47:03.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-03T04:47:03.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-03T04:47:03.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-06-03T04:47:03.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-06-03T04:47:03.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-06-03T04:47:03.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-06-03T04:47:03.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-06-03T04:47:03.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-06-03T04:47:03.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/network-ping 2024-06-03T04:47:03.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:6: run: local dir=td/network-ping 2024-06-03T04:47:03.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:7: run: shift 2024-06-03T04:47:03.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:9: run: export CEPH_MON=127.0.0.1:7146 2024-06-03T04:47:03.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:9: run: CEPH_MON=127.0.0.1:7146 2024-06-03T04:47:03.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:10: run: export CEPH_ARGS 2024-06-03T04:47:03.410 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:11: run: uuidgen 2024-06-03T04:47:03.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:11: run: CEPH_ARGS+='--fsid=9ee5db8b-6e0e-4fdb-a603-cfc7f9259256 --auth-supported=none ' 2024-06-03T04:47:03.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:12: run: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-06-03T04:47:03.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:13: run: CEPH_ARGS+='--debug_disable_randomized_ping=true ' 2024-06-03T04:47:03.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:14: run: CEPH_ARGS+='--debug_heartbeat_testing_span=5 ' 2024-06-03T04:47:03.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:15: run: CEPH_ARGS+='--osd_heartbeat_interval=1 ' 2024-06-03T04:47:03.411 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: set 2024-06-03T04:47:03.411 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-06-03T04:47:03.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: local 'funcs=TEST_network_ping_test1 2024-06-03T04:47:03.413 INFO:tasks.workunit.client.0.smithi088.stderr:TEST_network_ping_test2' 2024-06-03T04:47:03.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:17: run: for func in $funcs 2024-06-03T04:47:03.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:18: run: setup td/network-ping 2024-06-03T04:47:03.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/network-ping 2024-06-03T04:47:03.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/network-ping 2024-06-03T04:47:03.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-06-03T04:47:03.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T04:47:03.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-06-03T04:47:03.414 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:47:03.414 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:47:03.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:47:03.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:47:03.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:47:03.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:47:03.415 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:47:03.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:47:03.416 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:47:03.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:47:03.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:47:03.417 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:47:03.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:47:03.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:47:03.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:47:03.418 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:47:03.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:47:03.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T04:47:03.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-06-03T04:47:03.421 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:47:03.421 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:47:03.421 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34744 2024-06-03T04:47:03.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.34744 2024-06-03T04:47:03.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:47:03.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:47:03.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/network-ping 2024-06-03T04:47:03.423 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T04:47:03.423 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:47:03.423 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34744 2024-06-03T04:47:03.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.34744 2024-06-03T04:47:03.425 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T04:47:03.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-06-03T04:47:03.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-06-03T04:47:03.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T04:47:03.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/network-ping 1' TERM HUP INT 2024-06-03T04:47:03.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:19: run: TEST_network_ping_test1 td/network-ping 2024-06-03T04:47:03.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:25: TEST_network_ping_test1: local dir=td/network-ping 2024-06-03T04:47:03.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:27: TEST_network_ping_test1: run_mon td/network-ping a 2024-06-03T04:47:03.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/network-ping 2024-06-03T04:47:03.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T04:47:03.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T04:47:03.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T04:47:03.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/network-ping/a 2024-06-03T04:47:03.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/network-ping/a --run-dir=td/network-ping 2024-06-03T04:47:03.863 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T04:47:03.863 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:47:03.863 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:47:03.864 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:47:03.864 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:47:03.864 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34744 2024-06-03T04:47:03.864 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34744/$cluster-$name.asok' 2024-06-03T04:47:03.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/network-ping/a '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.34744/$cluster-$name.asok' --mon-cluster-log-file=td/network-ping/log --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-03T04:47:04.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T04:47:04.057 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T04:47:04.058 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:47:04.058 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T04:47:04.058 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T04:47:04.059 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T04:47:04.059 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T04:47:04.059 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T04:47:04.059 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T04:47:04.059 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:47:04.059 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:47:04.059 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34744 2024-06-03T04:47:04.060 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34744/ceph-mon.a.asok 2024-06-03T04:47:04.060 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:47:04.060 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.34744/ceph-mon.a.asok config get fsid 2024-06-03T04:47:04.172 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T04:47:04.172 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:47:04.172 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T04:47:04.172 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T04:47:04.172 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T04:47:04.173 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T04:47:04.173 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T04:47:04.173 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T04:47:04.173 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:47:04.173 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:47:04.173 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34744 2024-06-03T04:47:04.173 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34744/ceph-mon.a.asok 2024-06-03T04:47:04.173 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:47:04.174 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.34744/ceph-mon.a.asok config get mon_host 2024-06-03T04:47:04.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:28: TEST_network_ping_test1: run_mgr td/network-ping x 2024-06-03T04:47:04.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/network-ping 2024-06-03T04:47:04.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T04:47:04.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T04:47:04.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T04:47:04.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/network-ping/x 2024-06-03T04:47:04.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T04:47:05.179 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T04:47:05.179 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:47:05.179 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:47:05.179 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:47:05.179 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:47:05.180 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34744 2024-06-03T04:47:05.180 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34744/$cluster-$name.asok' 2024-06-03T04:47:05.180 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T04:47:05.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/network-ping/x '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.34744/$cluster-$name.asok' --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T04:47:05.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:29: TEST_network_ping_test1: run_osd td/network-ping 0 2024-06-03T04:47:05.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-06-03T04:47:05.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T04:47:05.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T04:47:05.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T04:47:05.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/0 2024-06-03T04:47:05.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9ee5db8b-6e0e-4fdb-a603-cfc7f9259256 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-06-03T04:47:05.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T04:47:05.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T04:47:05.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T04:47:05.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/0' 2024-06-03T04:47:05.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/0/journal' 2024-06-03T04:47:05.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T04:47:05.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T04:47:05.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-06-03T04:47:05.219 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T04:47:05.219 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:47:05.219 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:47:05.219 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:47:05.220 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:47:05.220 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34744 2024-06-03T04:47:05.220 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34744/$cluster-$name.asok' 2024-06-03T04:47:05.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34744/$cluster-$name.asok' 2024-06-03T04:47:05.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T04:47:05.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T04:47:05.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T04:47:05.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-06-03T04:47:05.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-06-03T04:47:05.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T04:47:05.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T04:47:05.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T04:47:05.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T04:47:05.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T04:47:05.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T04:47:05.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/0 2024-06-03T04:47:05.221 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T04:47:05.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3b637db9-ac23-4f9e-8912-05592a2ff8da 2024-06-03T04:47:05.221 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 3b637db9-ac23-4f9e-8912-05592a2ff8da 2024-06-03T04:47:05.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3b637db9-ac23-4f9e-8912-05592a2ff8da' 2024-06-03T04:47:05.222 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T04:47:05.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDJSl1mSUQPDhAA31xTp4idh1esmmHVYD+9XQ== 2024-06-03T04:47:05.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDJSl1mSUQPDhAA31xTp4idh1esmmHVYD+9XQ=="}' 2024-06-03T04:47:05.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3b637db9-ac23-4f9e-8912-05592a2ff8da -i td/network-ping/0/new.json 2024-06-03T04:47:05.522 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T04:47:05.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/0/new.json 2024-06-03T04:47:05.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=9ee5db8b-6e0e-4fdb-a603-cfc7f9259256 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.34744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDJSl1mSUQPDhAA31xTp4idh1esmmHVYD+9XQ== --osd-uuid 3b637db9-ac23-4f9e-8912-05592a2ff8da 2024-06-03T04:47:05.576 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:47:05.569+0000 7f7c78eb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:47:05.589 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:47:05.581+0000 7f7c78eb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:47:05.591 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:47:05.585+0000 7f7c78eb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:47:05.591 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:47:05.585+0000 7f7c78eb27c0 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-06-03T04:47:05.591 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:47:05.585+0000 7f7c78eb27c0 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-06-03T04:47:05.591 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:47:05.585+0000 7f7c78eb27c0 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-06-03T04:47:05.621 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:47:05.613+0000 7f7c78eb27c0 -1 bluestore(td/network-ping/0) _read_fsid unparsable uuid 2024-06-03T04:47:11.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/0/keyring 2024-06-03T04:47:11.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T04:47:11.819 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T04:47:11.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T04:47:11.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T04:47:12.372 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T04:47:12.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T04:47:12.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=9ee5db8b-6e0e-4fdb-a603-cfc7f9259256 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.34744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T04:47:12.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T04:47:12.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T04:47:12.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T04:47:12.405 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:47:12.397+0000 7f470cf307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:47:12.407 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:47:12.401+0000 7f470cf307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:47:12.408 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:47:12.401+0000 7f470cf307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:47:12.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T04:47:12.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T04:47:12.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T04:47:12.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T04:47:12.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T04:47:12.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:47:12.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T04:47:12.842 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T04:47:12.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:47:12.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:47:12.975 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:47:12.969+0000 7f470cf307c0 -1 Falling back to public interface 2024-06-03T04:47:13.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:47:14.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:47:14.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:47:14.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T04:47:14.251 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T04:47:14.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:47:14.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:47:14.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:47:15.612 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:47:15.605+0000 7f470cf307c0 -1 osd.0 0 log_to_monitors true 2024-06-03T04:47:15.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:47:15.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:47:15.657 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T04:47:15.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T04:47:15.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:47:15.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:47:16.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:47:17.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:47:17.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:47:17.305 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T04:47:17.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T04:47:17.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:47:17.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:47:17.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:47:18.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:47:18.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:47:18.753 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T04:47:18.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T04:47:18.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:47:18.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:47:19.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:47:20.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:47:20.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:47:20.228 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T04:47:20.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T04:47:20.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:47:20.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:47:20.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:47:21.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:47:21.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:47:21.729 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T04:47:21.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T04:47:21.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:47:21.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:47:22.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:47:23.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:47:23.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:47:23.186 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T04:47:23.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T04:47:23.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:47:23.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:47:23.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:47:24.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:47:24.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:47:24.751 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T04:47:24.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T04:47:24.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:47:24.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:47:25.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:47:26.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:47:26.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:47:26.199 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T04:47:26.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T04:47:26.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:47:26.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:47:26.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:47:27.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:47:27.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:47:27.769 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T04:47:27.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T04:47:27.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:47:27.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:47:28.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:47:29.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:47:29.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:47:29.215 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T04:47:29.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T04:47:29.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:47:29.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:47:30.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:47:31.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:47:31.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:47:31.037 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T04:47:31.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T04:47:31.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:47:31.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:47:31.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:47:32.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:47:32.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:47:32.879 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T04:47:32.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T04:47:32.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:47:32.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:47:33.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:47:34.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:47:34.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:47:34.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T04:47:34.589 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T04:47:34.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:47:34.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:47:35.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:47:36.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:47:36.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:47:36.468 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T04:47:36.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T04:47:36.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:47:36.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:47:37.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:47:38.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:47:38.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:47:38.483 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T04:47:38.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T04:47:38.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:47:38.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:47:39.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:47:40.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:47:40.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:47:40.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T04:47:40.607 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T04:47:40.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:47:40.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:47:41.128 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1767825953,v1:127.0.0.1:6803/1767825953] [v2:127.0.0.1:6804/1767825953,v1:127.0.0.1:6805/1767825953] exists,up 3b637db9-ac23-4f9e-8912-05592a2ff8da 2024-06-03T04:47:41.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T04:47:41.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T04:47:41.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T04:47:41.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:30: TEST_network_ping_test1: run_osd td/network-ping 1 2024-06-03T04:47:41.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-06-03T04:47:41.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T04:47:41.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-03T04:47:41.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T04:47:41.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/1 2024-06-03T04:47:41.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9ee5db8b-6e0e-4fdb-a603-cfc7f9259256 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-06-03T04:47:41.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T04:47:41.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T04:47:41.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T04:47:41.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/1' 2024-06-03T04:47:41.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/1/journal' 2024-06-03T04:47:41.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T04:47:41.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T04:47:41.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-06-03T04:47:41.131 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T04:47:41.131 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:47:41.131 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:47:41.131 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:47:41.131 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:47:41.131 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34744 2024-06-03T04:47:41.132 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34744/$cluster-$name.asok' 2024-06-03T04:47:41.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34744/$cluster-$name.asok' 2024-06-03T04:47:41.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T04:47:41.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T04:47:41.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T04:47:41.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-06-03T04:47:41.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-06-03T04:47:41.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T04:47:41.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T04:47:41.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T04:47:41.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T04:47:41.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T04:47:41.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T04:47:41.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/1 2024-06-03T04:47:41.134 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T04:47:41.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b5263732-b4d2-4534-8547-28cd46a04bd0 2024-06-03T04:47:41.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 b5263732-b4d2-4534-8547-28cd46a04bd0' 2024-06-03T04:47:41.134 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 b5263732-b4d2-4534-8547-28cd46a04bd0 2024-06-03T04:47:41.135 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T04:47:41.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDtSl1mM2sdCRAAiw6ojcln6++wOrIjMo5aCw== 2024-06-03T04:47:41.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDtSl1mM2sdCRAAiw6ojcln6++wOrIjMo5aCw=="}' 2024-06-03T04:47:41.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b5263732-b4d2-4534-8547-28cd46a04bd0 -i td/network-ping/1/new.json 2024-06-03T04:47:41.679 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T04:47:41.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/1/new.json 2024-06-03T04:47:41.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=9ee5db8b-6e0e-4fdb-a603-cfc7f9259256 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.34744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDtSl1mM2sdCRAAiw6ojcln6++wOrIjMo5aCw== --osd-uuid b5263732-b4d2-4534-8547-28cd46a04bd0 2024-06-03T04:47:41.726 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:47:41.721+0000 7f7c72a1b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:47:41.790 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:47:41.785+0000 7f7c72a1b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:47:41.792 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:47:41.785+0000 7f7c72a1b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:47:41.792 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:47:41.785+0000 7f7c72a1b7c0 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-06-03T04:47:41.792 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:47:41.785+0000 7f7c72a1b7c0 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-06-03T04:47:41.792 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:47:41.785+0000 7f7c72a1b7c0 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-06-03T04:47:41.906 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:47:41.901+0000 7f7c72a1b7c0 -1 bluestore(td/network-ping/1) _read_fsid unparsable uuid 2024-06-03T04:47:48.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/1/keyring 2024-06-03T04:47:48.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T04:47:48.387 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-06-03T04:47:48.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-03T04:47:48.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T04:47:48.930 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-06-03T04:47:48.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-03T04:47:48.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=9ee5db8b-6e0e-4fdb-a603-cfc7f9259256 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.34744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T04:47:48.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T04:47:48.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T04:47:48.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T04:47:48.962 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:47:48.956+0000 7f73947937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:47:48.965 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:47:48.956+0000 7f73947937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:47:48.966 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:47:48.960+0000 7f73947937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:47:49.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-03T04:47:49.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T04:47:49.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T04:47:49.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T04:47:49.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T04:47:49.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:47:49.336 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T04:47:49.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T04:47:49.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:47:49.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:47:49.527 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:47:49.520+0000 7f73947937c0 -1 Falling back to public interface 2024-06-03T04:47:49.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:47:50.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:47:50.733 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T04:47:50.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:47:50.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T04:47:50.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:47:50.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:47:51.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:47:52.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:47:52.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:47:52.138 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T04:47:52.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T04:47:52.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:47:52.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:47:52.139 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:47:52.132+0000 7f73947937c0 -1 osd.1 0 log_to_monitors true 2024-06-03T04:47:52.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:47:53.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:47:53.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:47:53.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T04:47:53.702 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T04:47:53.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:47:53.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:47:54.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:47:55.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:47:55.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:47:55.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T04:47:55.262 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T04:47:55.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:47:55.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:47:55.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:47:56.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:47:56.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:47:56.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T04:47:56.709 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T04:47:56.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:47:56.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:47:57.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:47:58.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:47:58.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:47:58.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T04:47:58.212 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T04:47:58.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:47:58.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:47:58.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:47:59.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:47:59.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:47:59.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T04:47:59.774 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T04:47:59.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:47:59.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:48:00.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:48:01.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:48:01.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:48:01.303 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T04:48:01.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T04:48:01.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:48:01.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:48:01.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:48:02.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:48:02.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:48:02.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T04:48:02.726 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T04:48:02.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:48:02.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:48:03.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:48:04.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:48:04.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:48:04.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T04:48:04.174 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T04:48:04.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:48:04.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:48:04.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:48:05.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:48:05.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:48:05.706 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T04:48:05.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T04:48:05.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:48:05.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:48:06.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:48:07.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:48:07.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:48:07.704 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T04:48:07.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T04:48:07.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:48:07.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:48:08.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:48:09.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:48:09.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:48:09.492 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T04:48:09.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T04:48:09.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:48:09.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:48:10.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:48:11.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:48:11.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:48:11.600 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T04:48:11.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T04:48:11.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:48:11.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:48:12.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:48:13.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:48:13.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:48:13.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T04:48:13.970 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T04:48:13.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:48:13.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:48:14.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:48:15.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:48:15.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:48:15.754 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T04:48:15.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T04:48:15.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:48:15.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:48:16.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:48:17.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:48:17.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:48:17.612 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T04:48:17.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T04:48:17.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:48:17.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:48:18.033 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2694181935,v1:127.0.0.1:6811/2694181935] [v2:127.0.0.1:6812/2694181935,v1:127.0.0.1:6813/2694181935] exists,up b5263732-b4d2-4534-8547-28cd46a04bd0 2024-06-03T04:48:18.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T04:48:18.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T04:48:18.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T04:48:18.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:31: TEST_network_ping_test1: run_osd td/network-ping 2 2024-06-03T04:48:18.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-06-03T04:48:18.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T04:48:18.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-03T04:48:18.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T04:48:18.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/2 2024-06-03T04:48:18.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9ee5db8b-6e0e-4fdb-a603-cfc7f9259256 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-06-03T04:48:18.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T04:48:18.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T04:48:18.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T04:48:18.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/2' 2024-06-03T04:48:18.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/2/journal' 2024-06-03T04:48:18.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T04:48:18.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T04:48:18.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-06-03T04:48:18.038 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T04:48:18.038 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:48:18.038 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:48:18.038 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:48:18.038 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:48:18.038 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34744 2024-06-03T04:48:18.039 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34744/$cluster-$name.asok' 2024-06-03T04:48:18.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34744/$cluster-$name.asok' 2024-06-03T04:48:18.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T04:48:18.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T04:48:18.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T04:48:18.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-06-03T04:48:18.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-06-03T04:48:18.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T04:48:18.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T04:48:18.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T04:48:18.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T04:48:18.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T04:48:18.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T04:48:18.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/2 2024-06-03T04:48:18.040 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T04:48:18.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=98a90ca6-4051-491f-943e-c202f8324da4 2024-06-03T04:48:18.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 98a90ca6-4051-491f-943e-c202f8324da4' 2024-06-03T04:48:18.041 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 98a90ca6-4051-491f-943e-c202f8324da4 2024-06-03T04:48:18.041 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T04:48:18.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQASS11mJ9ODAxAA2UmXWIX6QwcssHr6VB6ixA== 2024-06-03T04:48:18.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQASS11mJ9ODAxAA2UmXWIX6QwcssHr6VB6ixA=="}' 2024-06-03T04:48:18.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 98a90ca6-4051-491f-943e-c202f8324da4 -i td/network-ping/2/new.json 2024-06-03T04:48:18.599 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T04:48:18.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/2/new.json 2024-06-03T04:48:18.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=9ee5db8b-6e0e-4fdb-a603-cfc7f9259256 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.34744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQASS11mJ9ODAxAA2UmXWIX6QwcssHr6VB6ixA== --osd-uuid 98a90ca6-4051-491f-943e-c202f8324da4 2024-06-03T04:48:18.650 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:48:18.648+0000 7fbe90e777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:48:18.736 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:48:18.732+0000 7fbe90e777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:48:18.744 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:48:18.732+0000 7fbe90e777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:48:18.745 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:48:18.736+0000 7fbe90e777c0 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-06-03T04:48:18.745 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:48:18.736+0000 7fbe90e777c0 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-06-03T04:48:18.745 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:48:18.736+0000 7fbe90e777c0 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-06-03T04:48:18.791 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:48:18.788+0000 7fbe90e777c0 -1 bluestore(td/network-ping/2) _read_fsid unparsable uuid 2024-06-03T04:48:25.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/2/keyring 2024-06-03T04:48:25.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T04:48:25.636 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-06-03T04:48:25.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-03T04:48:25.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T04:48:26.185 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-06-03T04:48:26.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-03T04:48:26.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=9ee5db8b-6e0e-4fdb-a603-cfc7f9259256 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.34744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T04:48:26.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T04:48:26.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T04:48:26.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T04:48:26.217 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:48:26.212+0000 7f84c7c1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:48:26.219 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:48:26.216+0000 7f84c7c1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:48:26.221 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:48:26.216+0000 7f84c7c1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:48:26.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-03T04:48:26.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T04:48:26.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T04:48:26.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T04:48:26.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T04:48:26.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:48:26.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T04:48:26.637 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T04:48:26.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:48:26.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:48:26.780 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:48:26.776+0000 7f84c7c1d7c0 -1 Falling back to public interface 2024-06-03T04:48:27.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:48:28.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:48:28.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:48:28.060 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T04:48:28.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T04:48:28.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:48:28.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:48:28.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:48:29.446 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:48:29.444+0000 7f84c7c1d7c0 -1 osd.2 0 log_to_monitors true 2024-06-03T04:48:29.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:48:29.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:48:29.521 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T04:48:29.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T04:48:29.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:48:29.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:48:30.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:48:31.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:48:31.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:48:31.098 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T04:48:31.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T04:48:31.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:48:31.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:48:31.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:48:32.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:48:32.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:48:32.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T04:48:32.654 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T04:48:32.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:48:32.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:48:33.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:48:34.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:48:34.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:48:34.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T04:48:34.062 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T04:48:34.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:48:34.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:48:34.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:48:35.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:48:35.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:48:35.708 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T04:48:35.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T04:48:35.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:48:35.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:48:36.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:48:37.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:48:37.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:48:37.132 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T04:48:37.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T04:48:37.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:48:37.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:48:37.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:48:38.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:48:38.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:48:38.615 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T04:48:38.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T04:48:38.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:48:38.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:48:39.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:48:40.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:48:40.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:48:40.072 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T04:48:40.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T04:48:40.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:48:40.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:48:40.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:48:41.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:48:41.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:48:41.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T04:48:41.535 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T04:48:41.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:48:41.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:48:41.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:48:42.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:48:42.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:48:42.965 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T04:48:42.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T04:48:42.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:48:42.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:48:43.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:48:44.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:48:44.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:48:44.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T04:48:44.378 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T04:48:44.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:48:44.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:48:44.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:48:45.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:48:45.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:48:45.872 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T04:48:45.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T04:48:45.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:48:45.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:48:47.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:48:48.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:48:48.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:48:48.193 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T04:48:48.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T04:48:48.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:48:48.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:48:48.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:48:49.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:48:49.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:48:49.979 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T04:48:49.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T04:48:49.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:48:49.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:48:50.779 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:48:51.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:48:51.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:48:51.782 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T04:48:51.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T04:48:51.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:48:51.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:48:52.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:48:53.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:48:53.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:48:53.772 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T04:48:53.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T04:48:53.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:48:53.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:48:54.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:48:55.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:48:55.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:48:55.722 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-06-03T04:48:55.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T04:48:55.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:48:55.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:48:56.182 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3723119298,v1:127.0.0.1:6819/3723119298] [v2:127.0.0.1:6820/3723119298,v1:127.0.0.1:6821/3723119298] exists,up 98a90ca6-4051-491f-943e-c202f8324da4 2024-06-03T04:48:56.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T04:48:56.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T04:48:56.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T04:48:56.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:33: TEST_network_ping_test1: sleep 5 2024-06-03T04:49:01.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:35: TEST_network_ping_test1: create_pool foo 16 2024-06-03T04:49:01.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-06-03T04:49:01.712 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'foo' already exists 2024-06-03T04:49:01.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T04:49:02.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:38: TEST_network_ping_test1: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2024-06-03T04:49:02.803 INFO:tasks.workunit.client.0.smithi088.stdout:hints = 1 2024-06-03T04:49:02.803 INFO:tasks.workunit.client.0.smithi088.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2024-06-03T04:49:02.803 INFO:tasks.workunit.client.0.smithi088.stdout:Object prefix: benchmark_data_smithi088_40258 2024-06-03T04:49:02.803 INFO:tasks.workunit.client.0.smithi088.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-03T04:49:02.803 INFO:tasks.workunit.client.0.smithi088.stdout: 0 0 0 0 0 0 - 0 2024-06-03T04:49:03.803 INFO:tasks.workunit.client.0.smithi088.stdout: 1 16 60 44 0.171855 0.171875 0.30024 0.31292 2024-06-03T04:49:04.803 INFO:tasks.workunit.client.0.smithi088.stdout: 2 16 111 95 0.185523 0.199219 0.274922 0.305214 2024-06-03T04:49:05.803 INFO:tasks.workunit.client.0.smithi088.stdout: 3 16 172 156 0.203101 0.238281 0.268413 0.295586 2024-06-03T04:49:06.803 INFO:tasks.workunit.client.0.smithi088.stdout: 4 16 223 207 0.202125 0.199219 0.309037 0.294666 2024-06-03T04:49:07.804 INFO:tasks.workunit.client.0.smithi088.stdout: 5 16 284 268 0.20935 0.238281 0.284718 0.290523 2024-06-03T04:49:08.804 INFO:tasks.workunit.client.0.smithi088.stdout: 6 16 335 319 0.207656 0.199219 0.292131 0.290163 2024-06-03T04:49:09.804 INFO:tasks.workunit.client.0.smithi088.stdout: 7 16 396 380 0.212028 0.238281 0.285063 0.289629 2024-06-03T04:49:10.804 INFO:tasks.workunit.client.0.smithi088.stdout: 8 16 458 442 0.215795 0.242188 0.276006 0.286645 2024-06-03T04:49:11.804 INFO:tasks.workunit.client.0.smithi088.stdout: 9 16 508 492 0.213515 0.195312 0.25693 0.288916 2024-06-03T04:49:12.804 INFO:tasks.workunit.client.0.smithi088.stdout: 10 16 559 543 0.212083 0.199219 0.313252 0.288266 2024-06-03T04:49:13.804 INFO:tasks.workunit.client.0.smithi088.stdout:Total time run: 10.1355 2024-06-03T04:49:13.805 INFO:tasks.workunit.client.0.smithi088.stdout:Total writes made: 559 2024-06-03T04:49:13.805 INFO:tasks.workunit.client.0.smithi088.stdout:Write size: 4096 2024-06-03T04:49:13.805 INFO:tasks.workunit.client.0.smithi088.stdout:Object size: 4096 2024-06-03T04:49:13.805 INFO:tasks.workunit.client.0.smithi088.stdout:Bandwidth (MB/sec): 0.21544 2024-06-03T04:49:13.805 INFO:tasks.workunit.client.0.smithi088.stdout:Stddev Bandwidth: 0.0247772 2024-06-03T04:49:13.805 INFO:tasks.workunit.client.0.smithi088.stdout:Max bandwidth (MB/sec): 0.242188 2024-06-03T04:49:13.805 INFO:tasks.workunit.client.0.smithi088.stdout:Min bandwidth (MB/sec): 0.171875 2024-06-03T04:49:13.805 INFO:tasks.workunit.client.0.smithi088.stdout:Average IOPS: 55 2024-06-03T04:49:13.805 INFO:tasks.workunit.client.0.smithi088.stdout:Stddev IOPS: 6.34298 2024-06-03T04:49:13.806 INFO:tasks.workunit.client.0.smithi088.stdout:Max IOPS: 62 2024-06-03T04:49:13.806 INFO:tasks.workunit.client.0.smithi088.stdout:Min IOPS: 44 2024-06-03T04:49:13.806 INFO:tasks.workunit.client.0.smithi088.stdout:Average Latency(s): 0.288319 2024-06-03T04:49:13.806 INFO:tasks.workunit.client.0.smithi088.stdout:Stddev Latency(s): 0.035434 2024-06-03T04:49:13.806 INFO:tasks.workunit.client.0.smithi088.stdout:Max latency(s): 0.525814 2024-06-03T04:49:13.806 INFO:tasks.workunit.client.0.smithi088.stdout:Min latency(s): 0.231665 2024-06-03T04:49:13.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:41: TEST_network_ping_test1: sleep 10 2024-06-03T04:49:23.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:42: TEST_network_ping_test1: flush_pg_stats 2024-06-03T04:49:23.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T04:49:23.939 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T04:49:24.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T04:49:24.346 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-06-03T04:49:24.346 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-06-03T04:49:24.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T04:49:24.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T04:49:24.346 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T04:49:24.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836508 2024-06-03T04:49:24.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836508 2024-06-03T04:49:24.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508' 2024-06-03T04:49:24.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T04:49:24.529 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T04:49:24.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672981 2024-06-03T04:49:24.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672981 2024-06-03T04:49:24.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672981' 2024-06-03T04:49:24.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T04:49:24.686 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T04:49:24.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542156 2024-06-03T04:49:24.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542156 2024-06-03T04:49:24.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672981 2-60129542156' 2024-06-03T04:49:24.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T04:49:24.837 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836508 2024-06-03T04:49:24.837 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T04:49:24.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T04:49:24.839 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836508 2024-06-03T04:49:24.839 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T04:49:24.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836508 2024-06-03T04:49:24.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-06-03T04:49:24.840 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836508 2024-06-03T04:49:24.840 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T04:49:25.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836508 2024-06-03T04:49:25.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T04:49:26.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T04:49:26.239 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T04:49:26.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836508 2024-06-03T04:49:26.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T04:49:26.716 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672981 2024-06-03T04:49:26.716 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T04:49:26.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T04:49:26.718 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672981 2024-06-03T04:49:26.719 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T04:49:26.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672981 2024-06-03T04:49:26.722 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672981 2024-06-03T04:49:26.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-06-03T04:49:26.722 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T04:49:27.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672981 -lt 42949672981 2024-06-03T04:49:27.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T04:49:27.123 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-60129542156 2024-06-03T04:49:27.123 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T04:49:27.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T04:49:27.126 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-60129542156 2024-06-03T04:49:27.126 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T04:49:27.127 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542156 2024-06-03T04:49:27.127 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 60129542156 2024-06-03T04:49:27.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 60129542156' 2024-06-03T04:49:27.128 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T04:49:27.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542156 -lt 60129542156 2024-06-03T04:49:27.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: tee td/network-ping/json 2024-06-03T04:49:27.534 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: get_asok_path osd.0 2024-06-03T04:49:27.534 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:49:27.535 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:49:27.535 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:49:27.535 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:49:27.535 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34744 2024-06-03T04:49:27.535 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34744/ceph-osd.0.asok 2024-06-03T04:49:27.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: CEPH_ARGS= 2024-06-03T04:49:27.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34744/ceph-osd.0.asok dump_osd_network 2024-06-03T04:49:27.636 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-06-03T04:49:27.636 INFO:tasks.workunit.client.0.smithi088.stdout: "threshold": 1000, 2024-06-03T04:49:27.636 INFO:tasks.workunit.client.0.smithi088.stdout: "entries": [] 2024-06-03T04:49:27.636 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-06-03T04:49:27.649 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: cat td/network-ping/json 2024-06-03T04:49:27.650 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: jq '.entries | length' 2024-06-03T04:49:27.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: test 0 = 0 2024-06-03T04:49:27.682 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: cat td/network-ping/json 2024-06-03T04:49:27.683 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: jq .threshold 2024-06-03T04:49:27.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: test 1000 = 1000 2024-06-03T04:49:27.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: tee td/network-ping/json 2024-06-03T04:49:27.716 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: get_asok_path mgr.x 2024-06-03T04:49:27.716 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-06-03T04:49:27.716 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-06-03T04:49:27.717 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:49:27.717 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:49:27.717 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34744 2024-06-03T04:49:27.717 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34744/ceph-mgr.x.asok 2024-06-03T04:49:27.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: CEPH_ARGS= 2024-06-03T04:49:27.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34744/ceph-mgr.x.asok dump_osd_network 2024-06-03T04:49:27.812 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-06-03T04:49:27.812 INFO:tasks.workunit.client.0.smithi088.stdout: "threshold": 1000, 2024-06-03T04:49:27.812 INFO:tasks.workunit.client.0.smithi088.stdout: "entries": [] 2024-06-03T04:49:27.812 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-06-03T04:49:27.827 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: cat td/network-ping/json 2024-06-03T04:49:27.827 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: jq '.entries | length' 2024-06-03T04:49:27.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: test 0 = 0 2024-06-03T04:49:27.864 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: cat td/network-ping/json 2024-06-03T04:49:27.864 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: jq .threshold 2024-06-03T04:49:27.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: test 1000 = 1000 2024-06-03T04:49:27.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: tee td/network-ping/json 2024-06-03T04:49:27.901 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: get_asok_path osd.0 2024-06-03T04:49:27.901 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:49:27.901 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:49:27.902 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:49:27.902 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:49:27.903 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34744 2024-06-03T04:49:27.903 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34744/ceph-osd.0.asok 2024-06-03T04:49:27.903 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: CEPH_ARGS= 2024-06-03T04:49:27.903 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34744/ceph-osd.0.asok dump_osd_network 0 2024-06-03T04:49:28.008 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-06-03T04:49:28.008 INFO:tasks.workunit.client.0.smithi088.stdout: "threshold": 0, 2024-06-03T04:49:28.008 INFO:tasks.workunit.client.0.smithi088.stdout: "entries": [ 2024-06-03T04:49:28.009 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:49:28.009 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:49:26 2024", 2024-06-03T04:49:28.009 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:49:28.009 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 0, 2024-06-03T04:49:28.009 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 2, 2024-06-03T04:49:28.009 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "front", 2024-06-03T04:49:28.009 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:49:28.009 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.546, 2024-06-03T04:49:28.009 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.584, 2024-06-03T04:49:28.009 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.546 2024-06-03T04:49:28.010 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.010 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:49:28.010 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.674, 2024-06-03T04:49:28.010 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.681, 2024-06-03T04:49:28.010 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.681 2024-06-03T04:49:28.010 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.010 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:49:28.010 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.674, 2024-06-03T04:49:28.010 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.681, 2024-06-03T04:49:28.010 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.681 2024-06-03T04:49:28.010 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.010 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.451 2024-06-03T04:49:28.010 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.011 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:49:28.011 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:49:26 2024", 2024-06-03T04:49:28.011 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:49:28.011 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 0, 2024-06-03T04:49:28.011 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 2, 2024-06-03T04:49:28.011 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "back", 2024-06-03T04:49:28.011 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:49:28.011 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.475, 2024-06-03T04:49:28.011 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.522, 2024-06-03T04:49:28.011 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.557 2024-06-03T04:49:28.011 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.011 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:49:28.011 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.568, 2024-06-03T04:49:28.012 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.649, 2024-06-03T04:49:28.012 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.649 2024-06-03T04:49:28.012 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.012 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:49:28.012 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.568, 2024-06-03T04:49:28.012 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.649, 2024-06-03T04:49:28.012 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.649 2024-06-03T04:49:28.012 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.012 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.372 2024-06-03T04:49:28.012 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.012 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:49:28.012 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:49:27 2024", 2024-06-03T04:49:28.012 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:49:28.013 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 0, 2024-06-03T04:49:28.013 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 1, 2024-06-03T04:49:28.013 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "front", 2024-06-03T04:49:28.013 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:49:28.013 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.443, 2024-06-03T04:49:28.013 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.514, 2024-06-03T04:49:28.013 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.494 2024-06-03T04:49:28.013 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.013 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:49:28.013 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.526, 2024-06-03T04:49:28.013 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.639, 2024-06-03T04:49:28.013 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.674 2024-06-03T04:49:28.013 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.013 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:49:28.014 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.526, 2024-06-03T04:49:28.014 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.639, 2024-06-03T04:49:28.014 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.674 2024-06-03T04:49:28.014 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.014 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.351 2024-06-03T04:49:28.014 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.014 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:49:28.014 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:49:27 2024", 2024-06-03T04:49:28.014 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:49:28.014 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 0, 2024-06-03T04:49:28.014 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 1, 2024-06-03T04:49:28.014 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "back", 2024-06-03T04:49:28.015 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:49:28.015 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.477, 2024-06-03T04:49:28.015 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.497, 2024-06-03T04:49:28.015 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.477 2024-06-03T04:49:28.015 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.015 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:49:28.015 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.719, 2024-06-03T04:49:28.015 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.719, 2024-06-03T04:49:28.015 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.719 2024-06-03T04:49:28.015 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.015 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:49:28.015 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.719, 2024-06-03T04:49:28.015 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.719, 2024-06-03T04:49:28.015 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.719 2024-06-03T04:49:28.015 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.016 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.391 2024-06-03T04:49:28.016 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T04:49:28.016 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-06-03T04:49:28.016 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-06-03T04:49:28.021 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: cat td/network-ping/json 2024-06-03T04:49:28.021 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: jq '.entries | length' 2024-06-03T04:49:28.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: test 4 = 4 2024-06-03T04:49:28.057 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: cat td/network-ping/json 2024-06-03T04:49:28.057 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: jq .threshold 2024-06-03T04:49:28.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: test 0 = 0 2024-06-03T04:49:28.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: tee td/network-ping/json 2024-06-03T04:49:28.087 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: get_asok_path mgr.x 2024-06-03T04:49:28.087 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-06-03T04:49:28.087 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-06-03T04:49:28.087 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:49:28.088 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:49:28.088 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34744 2024-06-03T04:49:28.088 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34744/ceph-mgr.x.asok 2024-06-03T04:49:28.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: CEPH_ARGS= 2024-06-03T04:49:28.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34744/ceph-mgr.x.asok dump_osd_network 0 2024-06-03T04:49:28.188 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-06-03T04:49:28.188 INFO:tasks.workunit.client.0.smithi088.stdout: "threshold": 0, 2024-06-03T04:49:28.189 INFO:tasks.workunit.client.0.smithi088.stdout: "entries": [ 2024-06-03T04:49:28.189 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:49:28.189 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:49:21 2024", 2024-06-03T04:49:28.189 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:49:28.189 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 0, 2024-06-03T04:49:28.189 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 2, 2024-06-03T04:49:28.189 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "front", 2024-06-03T04:49:28.189 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:49:28.189 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.599, 2024-06-03T04:49:28.189 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.580, 2024-06-03T04:49:28.189 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.545 2024-06-03T04:49:28.189 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.189 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:49:28.190 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.560, 2024-06-03T04:49:28.190 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.493, 2024-06-03T04:49:28.190 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.493 2024-06-03T04:49:28.190 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.190 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:49:28.190 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.654, 2024-06-03T04:49:28.190 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.681, 2024-06-03T04:49:28.190 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.681 2024-06-03T04:49:28.190 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.190 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.607 2024-06-03T04:49:28.190 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.190 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:49:28.190 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:49:25 2024", 2024-06-03T04:49:28.190 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:49:28.190 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 2, 2024-06-03T04:49:28.191 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 1, 2024-06-03T04:49:28.191 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "back", 2024-06-03T04:49:28.191 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:49:28.191 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.496, 2024-06-03T04:49:28.191 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.533, 2024-06-03T04:49:28.191 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.593 2024-06-03T04:49:28.191 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.191 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:49:28.191 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.327, 2024-06-03T04:49:28.191 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.327, 2024-06-03T04:49:28.191 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.327 2024-06-03T04:49:28.191 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.191 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:49:28.191 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.659, 2024-06-03T04:49:28.192 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.750, 2024-06-03T04:49:28.192 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.836 2024-06-03T04:49:28.192 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.192 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.528 2024-06-03T04:49:28.192 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.192 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:49:28.192 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:49:25 2024", 2024-06-03T04:49:28.192 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:49:28.192 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 2, 2024-06-03T04:49:28.192 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 1, 2024-06-03T04:49:28.192 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "front", 2024-06-03T04:49:28.192 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:49:28.192 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.533, 2024-06-03T04:49:28.192 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.572, 2024-06-03T04:49:28.193 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.590 2024-06-03T04:49:28.193 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.193 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:49:28.193 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.303, 2024-06-03T04:49:28.193 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.303, 2024-06-03T04:49:28.193 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.303 2024-06-03T04:49:28.193 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.193 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:49:28.193 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.654, 2024-06-03T04:49:28.193 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.662, 2024-06-03T04:49:28.193 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.667 2024-06-03T04:49:28.193 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.193 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.614 2024-06-03T04:49:28.193 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.194 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:49:28.194 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:49:21 2024", 2024-06-03T04:49:28.194 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:49:28.194 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 1, 2024-06-03T04:49:28.194 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 2, 2024-06-03T04:49:28.194 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "back", 2024-06-03T04:49:28.194 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:49:28.194 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.571, 2024-06-03T04:49:28.194 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.588, 2024-06-03T04:49:28.194 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.585 2024-06-03T04:49:28.194 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.194 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:49:28.194 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.460, 2024-06-03T04:49:28.195 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.460, 2024-06-03T04:49:28.195 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.460 2024-06-03T04:49:28.195 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.195 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:49:28.195 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.615, 2024-06-03T04:49:28.195 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.668, 2024-06-03T04:49:28.195 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.668 2024-06-03T04:49:28.195 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.195 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.539 2024-06-03T04:49:28.195 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.195 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:49:28.195 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:49:21 2024", 2024-06-03T04:49:28.195 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:49:28.195 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 1, 2024-06-03T04:49:28.196 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 2, 2024-06-03T04:49:28.196 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "front", 2024-06-03T04:49:28.196 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:49:28.196 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.562, 2024-06-03T04:49:28.196 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.549, 2024-06-03T04:49:28.196 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.565 2024-06-03T04:49:28.196 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.196 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:49:28.196 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.463, 2024-06-03T04:49:28.196 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.438, 2024-06-03T04:49:28.196 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.438 2024-06-03T04:49:28.196 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.196 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:49:28.196 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.654, 2024-06-03T04:49:28.197 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.664, 2024-06-03T04:49:28.197 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.664 2024-06-03T04:49:28.197 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.197 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.482 2024-06-03T04:49:28.197 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.197 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:49:28.197 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:49:21 2024", 2024-06-03T04:49:28.197 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:49:28.197 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 0, 2024-06-03T04:49:28.197 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 2, 2024-06-03T04:49:28.197 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "back", 2024-06-03T04:49:28.197 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:49:28.197 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.515, 2024-06-03T04:49:28.197 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.542, 2024-06-03T04:49:28.197 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.564 2024-06-03T04:49:28.198 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.198 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:49:28.198 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.483, 2024-06-03T04:49:28.198 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.483, 2024-06-03T04:49:28.198 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.483 2024-06-03T04:49:28.198 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.198 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:49:28.198 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.574, 2024-06-03T04:49:28.198 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.649, 2024-06-03T04:49:28.198 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.649 2024-06-03T04:49:28.198 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.198 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.568 2024-06-03T04:49:28.198 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.199 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:49:28.199 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:49:22 2024", 2024-06-03T04:49:28.199 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:49:28.199 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 0, 2024-06-03T04:49:28.199 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 1, 2024-06-03T04:49:28.199 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "front", 2024-06-03T04:49:28.199 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:49:28.199 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.551, 2024-06-03T04:49:28.199 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.537, 2024-06-03T04:49:28.199 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.492 2024-06-03T04:49:28.199 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.199 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:49:28.199 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.521, 2024-06-03T04:49:28.199 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.447, 2024-06-03T04:49:28.200 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.266 2024-06-03T04:49:28.200 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.200 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:49:28.200 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.639, 2024-06-03T04:49:28.200 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.639, 2024-06-03T04:49:28.200 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.674 2024-06-03T04:49:28.200 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.200 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.526 2024-06-03T04:49:28.200 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.200 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:49:28.200 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:49:23 2024", 2024-06-03T04:49:28.200 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:49:28.200 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 1, 2024-06-03T04:49:28.200 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 0, 2024-06-03T04:49:28.201 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "front", 2024-06-03T04:49:28.201 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:49:28.201 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.526, 2024-06-03T04:49:28.201 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.540, 2024-06-03T04:49:28.201 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.509 2024-06-03T04:49:28.201 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.201 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:49:28.201 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.510, 2024-06-03T04:49:28.201 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.465, 2024-06-03T04:49:28.201 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.287 2024-06-03T04:49:28.201 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.201 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:49:28.201 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.545, 2024-06-03T04:49:28.202 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.660, 2024-06-03T04:49:28.202 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.660 2024-06-03T04:49:28.202 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.202 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.636 2024-06-03T04:49:28.202 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.202 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:49:28.202 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:49:25 2024", 2024-06-03T04:49:28.202 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:49:28.202 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 2, 2024-06-03T04:49:28.202 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 0, 2024-06-03T04:49:28.202 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "front", 2024-06-03T04:49:28.202 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:49:28.202 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.450, 2024-06-03T04:49:28.202 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.526, 2024-06-03T04:49:28.203 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.534 2024-06-03T04:49:28.203 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.203 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:49:28.203 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.280, 2024-06-03T04:49:28.203 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.280, 2024-06-03T04:49:28.203 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.280 2024-06-03T04:49:28.203 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.203 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:49:28.203 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.563, 2024-06-03T04:49:28.203 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.637, 2024-06-03T04:49:28.203 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.637 2024-06-03T04:49:28.203 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.203 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.563 2024-06-03T04:49:28.203 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.204 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:49:28.204 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:49:25 2024", 2024-06-03T04:49:28.204 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:49:28.204 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 2, 2024-06-03T04:49:28.204 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 0, 2024-06-03T04:49:28.204 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "back", 2024-06-03T04:49:28.204 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:49:28.204 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.500, 2024-06-03T04:49:28.204 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.525, 2024-06-03T04:49:28.204 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.495 2024-06-03T04:49:28.204 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.204 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:49:28.204 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.351, 2024-06-03T04:49:28.204 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.351, 2024-06-03T04:49:28.205 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.351 2024-06-03T04:49:28.205 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.205 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:49:28.205 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.607, 2024-06-03T04:49:28.205 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.674, 2024-06-03T04:49:28.205 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.674 2024-06-03T04:49:28.205 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.205 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.481 2024-06-03T04:49:28.205 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.205 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:49:28.205 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:49:22 2024", 2024-06-03T04:49:28.205 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:49:28.205 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 0, 2024-06-03T04:49:28.205 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 1, 2024-06-03T04:49:28.206 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "back", 2024-06-03T04:49:28.206 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:49:28.206 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.496, 2024-06-03T04:49:28.206 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.512, 2024-06-03T04:49:28.206 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.474 2024-06-03T04:49:28.206 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.206 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:49:28.206 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.440, 2024-06-03T04:49:28.206 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.426, 2024-06-03T04:49:28.206 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.208 2024-06-03T04:49:28.206 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.206 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:49:28.207 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.573, 2024-06-03T04:49:28.207 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.712, 2024-06-03T04:49:28.207 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.712 2024-06-03T04:49:28.207 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.207 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.719 2024-06-03T04:49:28.207 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.207 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:49:28.207 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:49:23 2024", 2024-06-03T04:49:28.207 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:49:28.208 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 1, 2024-06-03T04:49:28.208 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 0, 2024-06-03T04:49:28.208 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "back", 2024-06-03T04:49:28.208 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:49:28.208 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.467, 2024-06-03T04:49:28.208 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.484, 2024-06-03T04:49:28.208 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.471 2024-06-03T04:49:28.208 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.208 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:49:28.208 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.424, 2024-06-03T04:49:28.208 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.363, 2024-06-03T04:49:28.209 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.330 2024-06-03T04:49:28.209 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.209 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:49:28.209 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.502, 2024-06-03T04:49:28.209 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.693, 2024-06-03T04:49:28.209 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.693 2024-06-03T04:49:28.209 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:28.209 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.412 2024-06-03T04:49:28.209 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T04:49:28.209 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-06-03T04:49:28.210 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-06-03T04:49:28.210 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: cat td/network-ping/json 2024-06-03T04:49:28.210 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: jq '.entries | length' 2024-06-03T04:49:28.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: test 12 = 12 2024-06-03T04:49:28.238 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: cat td/network-ping/json 2024-06-03T04:49:28.238 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: jq .threshold 2024-06-03T04:49:28.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: test 0 = 0 2024-06-03T04:49:28.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:61: TEST_network_ping_test1: sleep 20 2024-06-03T04:49:48.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:62: TEST_network_ping_test1: flush_pg_stats 2024-06-03T04:49:48.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T04:49:48.272 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T04:49:48.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T04:49:48.692 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-06-03T04:49:48.692 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-06-03T04:49:48.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T04:49:48.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T04:49:48.693 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T04:49:48.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836515 2024-06-03T04:49:48.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836515 2024-06-03T04:49:48.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836515' 2024-06-03T04:49:48.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T04:49:48.851 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T04:49:48.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672988 2024-06-03T04:49:48.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672988 2024-06-03T04:49:48.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836515 1-42949672988' 2024-06-03T04:49:48.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T04:49:48.998 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T04:49:49.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542163 2024-06-03T04:49:49.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542163 2024-06-03T04:49:49.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836515 1-42949672988 2-60129542163' 2024-06-03T04:49:49.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T04:49:49.151 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T04:49:49.151 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836515 2024-06-03T04:49:49.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T04:49:49.152 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836515 2024-06-03T04:49:49.152 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T04:49:49.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836515 2024-06-03T04:49:49.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836515' 2024-06-03T04:49:49.153 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836515 2024-06-03T04:49:49.154 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T04:49:49.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836513 -lt 21474836515 2024-06-03T04:49:49.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T04:49:50.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T04:49:50.535 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T04:49:50.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836515 -lt 21474836515 2024-06-03T04:49:50.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T04:49:50.939 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672988 2024-06-03T04:49:50.939 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T04:49:50.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T04:49:50.941 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672988 2024-06-03T04:49:50.941 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T04:49:50.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672988 2024-06-03T04:49:50.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672988' 2024-06-03T04:49:50.943 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672988 2024-06-03T04:49:50.943 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T04:49:51.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672988 -lt 42949672988 2024-06-03T04:49:51.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T04:49:51.375 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-60129542163 2024-06-03T04:49:51.375 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T04:49:51.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T04:49:51.378 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-60129542163 2024-06-03T04:49:51.378 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T04:49:51.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542163 2024-06-03T04:49:51.379 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 60129542163 2024-06-03T04:49:51.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 60129542163' 2024-06-03T04:49:51.380 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T04:49:51.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542164 -lt 60129542163 2024-06-03T04:49:51.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: tee td/network-ping/json 2024-06-03T04:49:51.849 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: get_asok_path osd.0 2024-06-03T04:49:51.849 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:49:51.849 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:49:51.850 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:49:51.850 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:49:51.850 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34744 2024-06-03T04:49:51.850 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34744/ceph-osd.0.asok 2024-06-03T04:49:51.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: CEPH_ARGS= 2024-06-03T04:49:51.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34744/ceph-osd.0.asok dump_osd_network 2024-06-03T04:49:51.956 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-06-03T04:49:51.956 INFO:tasks.workunit.client.0.smithi088.stdout: "threshold": 1000, 2024-06-03T04:49:51.956 INFO:tasks.workunit.client.0.smithi088.stdout: "entries": [] 2024-06-03T04:49:51.956 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-06-03T04:49:51.969 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: cat td/network-ping/json 2024-06-03T04:49:51.969 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: jq '.entries | length' 2024-06-03T04:49:52.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: test 0 = 0 2024-06-03T04:49:52.003 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: cat td/network-ping/json 2024-06-03T04:49:52.003 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: jq .threshold 2024-06-03T04:49:52.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: test 1000 = 1000 2024-06-03T04:49:52.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: tee td/network-ping/json 2024-06-03T04:49:52.037 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: get_asok_path mgr.x 2024-06-03T04:49:52.037 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-06-03T04:49:52.037 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-06-03T04:49:52.038 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:49:52.038 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:49:52.038 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34744 2024-06-03T04:49:52.038 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34744/ceph-mgr.x.asok 2024-06-03T04:49:52.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: CEPH_ARGS= 2024-06-03T04:49:52.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34744/ceph-mgr.x.asok dump_osd_network 2024-06-03T04:49:52.135 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-06-03T04:49:52.135 INFO:tasks.workunit.client.0.smithi088.stdout: "threshold": 1000, 2024-06-03T04:49:52.135 INFO:tasks.workunit.client.0.smithi088.stdout: "entries": [] 2024-06-03T04:49:52.135 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-06-03T04:49:52.149 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: cat td/network-ping/json 2024-06-03T04:49:52.149 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: jq '.entries | length' 2024-06-03T04:49:52.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: test 0 = 0 2024-06-03T04:49:52.185 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: cat td/network-ping/json 2024-06-03T04:49:52.185 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: jq .threshold 2024-06-03T04:49:52.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: test 1000 = 1000 2024-06-03T04:49:52.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: tee td/network-ping/json 2024-06-03T04:49:52.223 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: get_asok_path osd.0 2024-06-03T04:49:52.223 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:49:52.223 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:49:52.224 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:49:52.224 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:49:52.224 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34744 2024-06-03T04:49:52.224 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34744/ceph-osd.0.asok 2024-06-03T04:49:52.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: CEPH_ARGS= 2024-06-03T04:49:52.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34744/ceph-osd.0.asok dump_osd_network 0 2024-06-03T04:49:52.331 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-06-03T04:49:52.331 INFO:tasks.workunit.client.0.smithi088.stdout: "threshold": 0, 2024-06-03T04:49:52.331 INFO:tasks.workunit.client.0.smithi088.stdout: "entries": [ 2024-06-03T04:49:52.331 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:49:52.331 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:49:51 2024", 2024-06-03T04:49:52.331 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:49:52.331 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 0, 2024-06-03T04:49:52.331 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 2, 2024-06-03T04:49:52.331 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "front", 2024-06-03T04:49:52.331 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:49:52.331 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.489, 2024-06-03T04:49:52.332 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.544, 2024-06-03T04:49:52.332 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.552 2024-06-03T04:49:52.332 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.332 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:49:52.332 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.628, 2024-06-03T04:49:52.332 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.681, 2024-06-03T04:49:52.332 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.681 2024-06-03T04:49:52.332 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.332 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:49:52.332 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.628, 2024-06-03T04:49:52.332 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.681, 2024-06-03T04:49:52.332 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.681 2024-06-03T04:49:52.332 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.333 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.572 2024-06-03T04:49:52.333 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.333 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:49:52.333 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:49:47 2024", 2024-06-03T04:49:52.333 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:49:52.333 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 0, 2024-06-03T04:49:52.333 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 1, 2024-06-03T04:49:52.333 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "back", 2024-06-03T04:49:52.333 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:49:52.333 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.546, 2024-06-03T04:49:52.333 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.489, 2024-06-03T04:49:52.333 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.491 2024-06-03T04:49:52.333 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.334 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:49:52.334 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.633, 2024-06-03T04:49:52.334 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.719, 2024-06-03T04:49:52.334 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.719 2024-06-03T04:49:52.334 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.334 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:49:52.334 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.633, 2024-06-03T04:49:52.334 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.719, 2024-06-03T04:49:52.334 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.719 2024-06-03T04:49:52.334 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.334 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.452 2024-06-03T04:49:52.334 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.334 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:49:52.335 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:49:51 2024", 2024-06-03T04:49:52.335 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:49:52.335 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 0, 2024-06-03T04:49:52.335 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 2, 2024-06-03T04:49:52.335 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "back", 2024-06-03T04:49:52.335 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:49:52.335 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.453, 2024-06-03T04:49:52.335 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.522, 2024-06-03T04:49:52.335 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.540 2024-06-03T04:49:52.335 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.335 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:49:52.335 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.587, 2024-06-03T04:49:52.335 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.683, 2024-06-03T04:49:52.335 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.683 2024-06-03T04:49:52.335 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.336 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:49:52.336 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.587, 2024-06-03T04:49:52.336 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.683, 2024-06-03T04:49:52.336 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.683 2024-06-03T04:49:52.336 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.336 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.496 2024-06-03T04:49:52.336 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.336 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:49:52.336 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:49:47 2024", 2024-06-03T04:49:52.336 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:49:52.336 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 0, 2024-06-03T04:49:52.336 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 1, 2024-06-03T04:49:52.336 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "front", 2024-06-03T04:49:52.336 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:49:52.337 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.485, 2024-06-03T04:49:52.337 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.518, 2024-06-03T04:49:52.337 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.520 2024-06-03T04:49:52.337 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.337 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:49:52.337 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.550, 2024-06-03T04:49:52.337 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.699, 2024-06-03T04:49:52.337 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.699 2024-06-03T04:49:52.337 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.337 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:49:52.337 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.550, 2024-06-03T04:49:52.337 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.699, 2024-06-03T04:49:52.337 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.699 2024-06-03T04:49:52.337 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.337 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.528 2024-06-03T04:49:52.338 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T04:49:52.338 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-06-03T04:49:52.338 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-06-03T04:49:52.346 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: cat td/network-ping/json 2024-06-03T04:49:52.346 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: jq '.entries | length' 2024-06-03T04:49:52.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: test 4 = 4 2024-06-03T04:49:52.382 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: cat td/network-ping/json 2024-06-03T04:49:52.382 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: jq .threshold 2024-06-03T04:49:52.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: test 0 = 0 2024-06-03T04:49:52.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: tee td/network-ping/json 2024-06-03T04:49:52.420 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: get_asok_path mgr.x 2024-06-03T04:49:52.421 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-06-03T04:49:52.421 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-06-03T04:49:52.421 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:49:52.422 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:49:52.422 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34744 2024-06-03T04:49:52.422 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34744/ceph-mgr.x.asok 2024-06-03T04:49:52.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: CEPH_ARGS= 2024-06-03T04:49:52.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34744/ceph-mgr.x.asok dump_osd_network 0 2024-06-03T04:49:52.518 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-06-03T04:49:52.518 INFO:tasks.workunit.client.0.smithi088.stdout: "threshold": 0, 2024-06-03T04:49:52.518 INFO:tasks.workunit.client.0.smithi088.stdout: "entries": [ 2024-06-03T04:49:52.518 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:49:52.518 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:49:46 2024", 2024-06-03T04:49:52.518 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:49:52.518 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 1, 2024-06-03T04:49:52.518 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 2, 2024-06-03T04:49:52.518 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "back", 2024-06-03T04:49:52.518 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:49:52.518 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.591, 2024-06-03T04:49:52.519 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.572, 2024-06-03T04:49:52.519 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.581 2024-06-03T04:49:52.519 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.519 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:49:52.519 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.547, 2024-06-03T04:49:52.519 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.341, 2024-06-03T04:49:52.519 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.341 2024-06-03T04:49:52.519 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.519 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:49:52.519 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.628, 2024-06-03T04:49:52.519 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.670, 2024-06-03T04:49:52.519 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.670 2024-06-03T04:49:52.519 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.519 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.652 2024-06-03T04:49:52.520 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.520 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:49:52.520 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:49:50 2024", 2024-06-03T04:49:52.520 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:49:52.520 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 2, 2024-06-03T04:49:52.520 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 1, 2024-06-03T04:49:52.520 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "front", 2024-06-03T04:49:52.520 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:49:52.520 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.564, 2024-06-03T04:49:52.520 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.583, 2024-06-03T04:49:52.520 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.585 2024-06-03T04:49:52.520 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.520 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:49:52.520 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.501, 2024-06-03T04:49:52.521 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.478, 2024-06-03T04:49:52.521 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.303 2024-06-03T04:49:52.521 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.521 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:49:52.521 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.604, 2024-06-03T04:49:52.521 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.671, 2024-06-03T04:49:52.521 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.671 2024-06-03T04:49:52.521 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.521 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.533 2024-06-03T04:49:52.521 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.521 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:49:52.521 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:49:46 2024", 2024-06-03T04:49:52.521 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:49:52.521 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 0, 2024-06-03T04:49:52.522 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 2, 2024-06-03T04:49:52.522 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "back", 2024-06-03T04:49:52.522 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:49:52.523 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.563, 2024-06-03T04:49:52.523 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.527, 2024-06-03T04:49:52.523 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.548 2024-06-03T04:49:52.523 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.523 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:49:52.523 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.543, 2024-06-03T04:49:52.523 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.348, 2024-06-03T04:49:52.523 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.348 2024-06-03T04:49:52.523 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.523 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:49:52.523 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.609, 2024-06-03T04:49:52.523 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.683, 2024-06-03T04:49:52.523 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.683 2024-06-03T04:49:52.523 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.524 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.254 2024-06-03T04:49:52.524 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.524 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:49:52.524 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:49:48 2024", 2024-06-03T04:49:52.524 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:49:52.524 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 1, 2024-06-03T04:49:52.524 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 0, 2024-06-03T04:49:52.524 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "front", 2024-06-03T04:49:52.524 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:49:52.524 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.558, 2024-06-03T04:49:52.524 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.548, 2024-06-03T04:49:52.524 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.528 2024-06-03T04:49:52.524 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.525 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:49:52.525 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.476, 2024-06-03T04:49:52.525 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.285, 2024-06-03T04:49:52.525 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.285 2024-06-03T04:49:52.525 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.525 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:49:52.525 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.640, 2024-06-03T04:49:52.525 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.659, 2024-06-03T04:49:52.525 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.660 2024-06-03T04:49:52.525 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.525 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.572 2024-06-03T04:49:52.525 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.525 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:49:52.525 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:49:46 2024", 2024-06-03T04:49:52.525 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:49:52.526 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 0, 2024-06-03T04:49:52.526 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 2, 2024-06-03T04:49:52.526 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "front", 2024-06-03T04:49:52.526 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:49:52.526 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.515, 2024-06-03T04:49:52.526 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.556, 2024-06-03T04:49:52.526 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.554 2024-06-03T04:49:52.526 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.526 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:49:52.526 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.484, 2024-06-03T04:49:52.526 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.321, 2024-06-03T04:49:52.526 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.321 2024-06-03T04:49:52.527 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.527 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:49:52.527 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.582, 2024-06-03T04:49:52.527 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.681, 2024-06-03T04:49:52.527 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.681 2024-06-03T04:49:52.527 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.527 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.238 2024-06-03T04:49:52.527 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.527 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:49:52.527 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:49:50 2024", 2024-06-03T04:49:52.528 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:49:52.528 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 2, 2024-06-03T04:49:52.528 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 1, 2024-06-03T04:49:52.528 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "back", 2024-06-03T04:49:52.528 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:49:52.528 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.509, 2024-06-03T04:49:52.528 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.504, 2024-06-03T04:49:52.528 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.553 2024-06-03T04:49:52.528 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.528 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:49:52.528 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.421, 2024-06-03T04:49:52.528 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.310, 2024-06-03T04:49:52.528 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.310 2024-06-03T04:49:52.529 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.529 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:49:52.529 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.558, 2024-06-03T04:49:52.529 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.570, 2024-06-03T04:49:52.529 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.836 2024-06-03T04:49:52.529 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.529 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.558 2024-06-03T04:49:52.529 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.529 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:49:52.529 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:49:46 2024", 2024-06-03T04:49:52.529 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:49:52.529 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 1, 2024-06-03T04:49:52.529 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 2, 2024-06-03T04:49:52.530 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "front", 2024-06-03T04:49:52.530 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:49:52.530 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.532, 2024-06-03T04:49:52.530 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.536, 2024-06-03T04:49:52.530 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.553 2024-06-03T04:49:52.530 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.530 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:49:52.530 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.507, 2024-06-03T04:49:52.530 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.401, 2024-06-03T04:49:52.530 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.401 2024-06-03T04:49:52.530 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.530 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:49:52.530 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.597, 2024-06-03T04:49:52.530 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.698, 2024-06-03T04:49:52.531 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.698 2024-06-03T04:49:52.531 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.531 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.537 2024-06-03T04:49:52.531 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.531 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:49:52.531 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:49:47 2024", 2024-06-03T04:49:52.531 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:49:52.531 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 0, 2024-06-03T04:49:52.531 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 1, 2024-06-03T04:49:52.531 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "back", 2024-06-03T04:49:52.531 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:49:52.531 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.546, 2024-06-03T04:49:52.531 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.489, 2024-06-03T04:49:52.531 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.491 2024-06-03T04:49:52.531 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.532 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:49:52.532 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.434, 2024-06-03T04:49:52.532 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.372, 2024-06-03T04:49:52.532 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.363 2024-06-03T04:49:52.532 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.532 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:49:52.532 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.633, 2024-06-03T04:49:52.532 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.719, 2024-06-03T04:49:52.532 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.719 2024-06-03T04:49:52.532 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.532 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.280 2024-06-03T04:49:52.532 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.532 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:49:52.532 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:49:50 2024", 2024-06-03T04:49:52.533 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:49:52.533 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 2, 2024-06-03T04:49:52.533 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 0, 2024-06-03T04:49:52.533 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "front", 2024-06-03T04:49:52.533 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:49:52.533 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.503, 2024-06-03T04:49:52.533 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.519, 2024-06-03T04:49:52.533 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.527 2024-06-03T04:49:52.533 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.533 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:49:52.534 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.447, 2024-06-03T04:49:52.534 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.387, 2024-06-03T04:49:52.534 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.280 2024-06-03T04:49:52.534 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.534 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:49:52.534 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.558, 2024-06-03T04:49:52.534 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.608, 2024-06-03T04:49:52.534 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.637 2024-06-03T04:49:52.534 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.534 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.472 2024-06-03T04:49:52.534 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.534 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:49:52.534 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:49:47 2024", 2024-06-03T04:49:52.534 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:49:52.535 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 0, 2024-06-03T04:49:52.535 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 1, 2024-06-03T04:49:52.535 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "front", 2024-06-03T04:49:52.535 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:49:52.535 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.485, 2024-06-03T04:49:52.535 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.518, 2024-06-03T04:49:52.535 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.520 2024-06-03T04:49:52.535 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.535 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:49:52.535 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.428, 2024-06-03T04:49:52.535 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.300, 2024-06-03T04:49:52.535 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.300 2024-06-03T04:49:52.535 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.535 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:49:52.536 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.550, 2024-06-03T04:49:52.536 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.699, 2024-06-03T04:49:52.536 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.699 2024-06-03T04:49:52.536 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.536 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.222 2024-06-03T04:49:52.536 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.536 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:49:52.536 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:49:50 2024", 2024-06-03T04:49:52.536 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:49:52.536 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 2, 2024-06-03T04:49:52.536 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 0, 2024-06-03T04:49:52.536 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "back", 2024-06-03T04:49:52.536 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:49:52.536 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.455, 2024-06-03T04:49:52.537 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.492, 2024-06-03T04:49:52.537 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.499 2024-06-03T04:49:52.537 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.537 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:49:52.537 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.424, 2024-06-03T04:49:52.537 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.390, 2024-06-03T04:49:52.537 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.351 2024-06-03T04:49:52.537 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.537 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:49:52.537 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.517, 2024-06-03T04:49:52.537 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.640, 2024-06-03T04:49:52.537 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.674 2024-06-03T04:49:52.537 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.537 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.437 2024-06-03T04:49:52.537 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.538 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:49:52.538 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:49:48 2024", 2024-06-03T04:49:52.538 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:49:52.538 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 1, 2024-06-03T04:49:52.538 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 0, 2024-06-03T04:49:52.538 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "back", 2024-06-03T04:49:52.538 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:49:52.538 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.463, 2024-06-03T04:49:52.538 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.490, 2024-06-03T04:49:52.538 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.485 2024-06-03T04:49:52.538 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.538 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:49:52.538 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.439, 2024-06-03T04:49:52.538 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.381, 2024-06-03T04:49:52.538 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.330 2024-06-03T04:49:52.539 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.539 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:49:52.539 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.507, 2024-06-03T04:49:52.539 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.589, 2024-06-03T04:49:52.539 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.693 2024-06-03T04:49:52.539 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:49:52.539 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.440 2024-06-03T04:49:52.539 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T04:49:52.539 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-06-03T04:49:52.539 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-06-03T04:49:52.540 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: cat td/network-ping/json 2024-06-03T04:49:52.540 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: jq '.entries | length' 2024-06-03T04:49:52.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: test 12 = 12 2024-06-03T04:49:52.562 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: cat td/network-ping/json 2024-06-03T04:49:52.562 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: jq .threshold 2024-06-03T04:49:52.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: test 0 = 0 2024-06-03T04:49:52.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:81: TEST_network_ping_test1: sleep 50 2024-06-03T04:50:42.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:82: TEST_network_ping_test1: flush_pg_stats 2024-06-03T04:50:42.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T04:50:42.599 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T04:50:43.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T04:50:43.023 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-06-03T04:50:43.023 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-06-03T04:50:43.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T04:50:43.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T04:50:43.023 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T04:50:43.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836528 2024-06-03T04:50:43.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836528 2024-06-03T04:50:43.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836528' 2024-06-03T04:50:43.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T04:50:43.211 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T04:50:43.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673001 2024-06-03T04:50:43.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673001 2024-06-03T04:50:43.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836528 1-42949673001' 2024-06-03T04:50:43.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T04:50:43.359 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T04:50:43.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542176 2024-06-03T04:50:43.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542176 2024-06-03T04:50:43.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836528 1-42949673001 2-60129542176' 2024-06-03T04:50:43.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T04:50:43.507 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836528 2024-06-03T04:50:43.508 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T04:50:43.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T04:50:43.509 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836528 2024-06-03T04:50:43.509 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T04:50:43.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836528 2024-06-03T04:50:43.510 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836528 2024-06-03T04:50:43.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836528' 2024-06-03T04:50:43.510 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T04:50:43.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836528 -lt 21474836528 2024-06-03T04:50:43.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T04:50:43.914 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673001 2024-06-03T04:50:43.914 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T04:50:43.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T04:50:43.916 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673001 2024-06-03T04:50:43.916 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T04:50:43.918 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949673001 2024-06-03T04:50:43.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673001 2024-06-03T04:50:43.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673001' 2024-06-03T04:50:43.918 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T04:50:44.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673001 -lt 42949673001 2024-06-03T04:50:44.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T04:50:44.351 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-60129542176 2024-06-03T04:50:44.351 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T04:50:44.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T04:50:44.354 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-60129542176 2024-06-03T04:50:44.354 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T04:50:44.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542176 2024-06-03T04:50:44.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 60129542176' 2024-06-03T04:50:44.355 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 60129542176 2024-06-03T04:50:44.355 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T04:50:44.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542176 -lt 60129542176 2024-06-03T04:50:44.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: tee td/network-ping/json 2024-06-03T04:50:44.777 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: get_asok_path osd.0 2024-06-03T04:50:44.777 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:50:44.777 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:50:44.778 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:50:44.778 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:50:44.778 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34744 2024-06-03T04:50:44.778 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34744/ceph-osd.0.asok 2024-06-03T04:50:44.779 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: CEPH_ARGS= 2024-06-03T04:50:44.779 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34744/ceph-osd.0.asok dump_osd_network 2024-06-03T04:50:44.888 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-06-03T04:50:44.888 INFO:tasks.workunit.client.0.smithi088.stdout: "threshold": 1000, 2024-06-03T04:50:44.888 INFO:tasks.workunit.client.0.smithi088.stdout: "entries": [] 2024-06-03T04:50:44.888 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-06-03T04:50:44.902 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: cat td/network-ping/json 2024-06-03T04:50:44.902 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: jq '.entries | length' 2024-06-03T04:50:44.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: test 0 = 0 2024-06-03T04:50:44.938 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: cat td/network-ping/json 2024-06-03T04:50:44.938 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: jq .threshold 2024-06-03T04:50:44.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: test 1000 = 1000 2024-06-03T04:50:44.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: tee td/network-ping/json 2024-06-03T04:50:44.973 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: get_asok_path mgr.x 2024-06-03T04:50:44.973 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-06-03T04:50:44.973 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-06-03T04:50:44.973 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:50:44.973 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:50:44.973 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34744 2024-06-03T04:50:44.973 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34744/ceph-mgr.x.asok 2024-06-03T04:50:44.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: CEPH_ARGS= 2024-06-03T04:50:44.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34744/ceph-mgr.x.asok dump_osd_network 2024-06-03T04:50:45.061 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-06-03T04:50:45.061 INFO:tasks.workunit.client.0.smithi088.stdout: "threshold": 1000, 2024-06-03T04:50:45.061 INFO:tasks.workunit.client.0.smithi088.stdout: "entries": [] 2024-06-03T04:50:45.062 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-06-03T04:50:45.074 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: cat td/network-ping/json 2024-06-03T04:50:45.074 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: jq '.entries | length' 2024-06-03T04:50:45.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: test 0 = 0 2024-06-03T04:50:45.107 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: cat td/network-ping/json 2024-06-03T04:50:45.108 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: jq .threshold 2024-06-03T04:50:45.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: test 1000 = 1000 2024-06-03T04:50:45.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: tee td/network-ping/json 2024-06-03T04:50:45.143 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: get_asok_path osd.0 2024-06-03T04:50:45.143 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:50:45.144 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:50:45.144 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:50:45.144 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:50:45.144 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34744 2024-06-03T04:50:45.144 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34744/ceph-osd.0.asok 2024-06-03T04:50:45.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: CEPH_ARGS= 2024-06-03T04:50:45.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34744/ceph-osd.0.asok dump_osd_network 0 2024-06-03T04:50:45.248 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-06-03T04:50:45.248 INFO:tasks.workunit.client.0.smithi088.stdout: "threshold": 0, 2024-06-03T04:50:45.248 INFO:tasks.workunit.client.0.smithi088.stdout: "entries": [ 2024-06-03T04:50:45.249 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:50:45.249 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:50:41 2024", 2024-06-03T04:50:45.249 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:50:45.249 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 0, 2024-06-03T04:50:45.249 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 2, 2024-06-03T04:50:45.249 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "front", 2024-06-03T04:50:45.249 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:50:45.249 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.580, 2024-06-03T04:50:45.249 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.560, 2024-06-03T04:50:45.249 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.559 2024-06-03T04:50:45.249 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.249 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:50:45.249 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.630, 2024-06-03T04:50:45.249 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.674, 2024-06-03T04:50:45.250 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.681 2024-06-03T04:50:45.250 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.250 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:50:45.250 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.630, 2024-06-03T04:50:45.250 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.674, 2024-06-03T04:50:45.250 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.681 2024-06-03T04:50:45.250 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.250 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.436 2024-06-03T04:50:45.250 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.250 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:50:45.250 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:50:42 2024", 2024-06-03T04:50:45.250 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:50:45.250 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 0, 2024-06-03T04:50:45.251 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 1, 2024-06-03T04:50:45.251 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "back", 2024-06-03T04:50:45.251 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:50:45.251 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.487, 2024-06-03T04:50:45.251 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.539, 2024-06-03T04:50:45.251 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.508 2024-06-03T04:50:45.251 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.251 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:50:45.251 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.535, 2024-06-03T04:50:45.251 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.637, 2024-06-03T04:50:45.251 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.649 2024-06-03T04:50:45.251 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.251 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:50:45.251 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.535, 2024-06-03T04:50:45.252 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.637, 2024-06-03T04:50:45.252 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.649 2024-06-03T04:50:45.252 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.252 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.336 2024-06-03T04:50:45.252 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.252 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:50:45.252 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:50:42 2024", 2024-06-03T04:50:45.252 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:50:45.252 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 0, 2024-06-03T04:50:45.252 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 1, 2024-06-03T04:50:45.252 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "front", 2024-06-03T04:50:45.252 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:50:45.252 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.534, 2024-06-03T04:50:45.252 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.517, 2024-06-03T04:50:45.252 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.517 2024-06-03T04:50:45.253 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.253 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:50:45.253 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.608, 2024-06-03T04:50:45.253 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.608, 2024-06-03T04:50:45.253 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.699 2024-06-03T04:50:45.253 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.253 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:50:45.253 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.608, 2024-06-03T04:50:45.253 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.608, 2024-06-03T04:50:45.253 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.699 2024-06-03T04:50:45.253 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.253 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.408 2024-06-03T04:50:45.253 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.253 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:50:45.254 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:50:41 2024", 2024-06-03T04:50:45.254 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:50:45.254 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 0, 2024-06-03T04:50:45.254 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 2, 2024-06-03T04:50:45.254 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "back", 2024-06-03T04:50:45.254 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:50:45.254 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.528, 2024-06-03T04:50:45.254 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.522, 2024-06-03T04:50:45.254 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.530 2024-06-03T04:50:45.254 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.254 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:50:45.254 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.660, 2024-06-03T04:50:45.254 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.660, 2024-06-03T04:50:45.254 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.683 2024-06-03T04:50:45.254 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.255 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:50:45.255 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.660, 2024-06-03T04:50:45.255 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.660, 2024-06-03T04:50:45.255 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.683 2024-06-03T04:50:45.255 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.255 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.384 2024-06-03T04:50:45.255 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T04:50:45.255 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-06-03T04:50:45.255 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-06-03T04:50:45.265 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: cat td/network-ping/json 2024-06-03T04:50:45.265 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: jq '.entries | length' 2024-06-03T04:50:45.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: test 4 = 4 2024-06-03T04:50:45.299 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: cat td/network-ping/json 2024-06-03T04:50:45.299 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: jq .threshold 2024-06-03T04:50:45.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: test 0 = 0 2024-06-03T04:50:45.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: tee td/network-ping/json 2024-06-03T04:50:45.335 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: get_asok_path mgr.x 2024-06-03T04:50:45.335 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-06-03T04:50:45.335 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-06-03T04:50:45.336 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:50:45.336 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:50:45.336 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34744 2024-06-03T04:50:45.336 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34744/ceph-mgr.x.asok 2024-06-03T04:50:45.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: CEPH_ARGS= 2024-06-03T04:50:45.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34744/ceph-mgr.x.asok dump_osd_network 0 2024-06-03T04:50:45.438 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-06-03T04:50:45.438 INFO:tasks.workunit.client.0.smithi088.stdout: "threshold": 0, 2024-06-03T04:50:45.439 INFO:tasks.workunit.client.0.smithi088.stdout: "entries": [ 2024-06-03T04:50:45.439 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:50:45.439 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:50:41 2024", 2024-06-03T04:50:45.439 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:50:45.439 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 1, 2024-06-03T04:50:45.439 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 2, 2024-06-03T04:50:45.439 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "back", 2024-06-03T04:50:45.439 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:50:45.439 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.570, 2024-06-03T04:50:45.439 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.596, 2024-06-03T04:50:45.439 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.581 2024-06-03T04:50:45.439 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.439 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:50:45.440 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.452, 2024-06-03T04:50:45.440 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.452, 2024-06-03T04:50:45.440 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.292 2024-06-03T04:50:45.440 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.440 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:50:45.440 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.672, 2024-06-03T04:50:45.440 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.698, 2024-06-03T04:50:45.440 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.698 2024-06-03T04:50:45.440 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.440 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.303 2024-06-03T04:50:45.440 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.440 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:50:45.440 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:50:41 2024", 2024-06-03T04:50:45.441 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:50:45.441 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 1, 2024-06-03T04:50:45.441 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 2, 2024-06-03T04:50:45.441 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "front", 2024-06-03T04:50:45.441 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:50:45.441 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.595, 2024-06-03T04:50:45.441 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.557, 2024-06-03T04:50:45.441 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.541 2024-06-03T04:50:45.441 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.441 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:50:45.441 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.490, 2024-06-03T04:50:45.441 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.466, 2024-06-03T04:50:45.441 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.287 2024-06-03T04:50:45.441 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.442 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:50:45.442 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.737, 2024-06-03T04:50:45.442 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.737, 2024-06-03T04:50:45.442 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.737 2024-06-03T04:50:45.442 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.442 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.280 2024-06-03T04:50:45.442 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.442 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:50:45.442 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:50:41 2024", 2024-06-03T04:50:45.442 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:50:45.442 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 0, 2024-06-03T04:50:45.442 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 2, 2024-06-03T04:50:45.442 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "front", 2024-06-03T04:50:45.443 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:50:45.443 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.580, 2024-06-03T04:50:45.443 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.560, 2024-06-03T04:50:45.443 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.559 2024-06-03T04:50:45.443 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.443 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:50:45.443 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.539, 2024-06-03T04:50:45.443 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.448, 2024-06-03T04:50:45.443 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.238 2024-06-03T04:50:45.443 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.443 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:50:45.443 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.630, 2024-06-03T04:50:45.443 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.674, 2024-06-03T04:50:45.443 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.681 2024-06-03T04:50:45.444 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.444 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.545 2024-06-03T04:50:45.444 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.444 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:50:45.444 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:50:40 2024", 2024-06-03T04:50:45.444 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:50:45.444 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 2, 2024-06-03T04:50:45.444 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 1, 2024-06-03T04:50:45.444 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "front", 2024-06-03T04:50:45.444 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:50:45.444 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.578, 2024-06-03T04:50:45.445 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.571, 2024-06-03T04:50:45.445 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.565 2024-06-03T04:50:45.445 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.445 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:50:45.445 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.533, 2024-06-03T04:50:45.445 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.472, 2024-06-03T04:50:45.445 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.337 2024-06-03T04:50:45.445 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.445 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:50:45.445 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.611, 2024-06-03T04:50:45.445 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.667, 2024-06-03T04:50:45.445 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.671 2024-06-03T04:50:45.445 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.446 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.275 2024-06-03T04:50:45.446 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.446 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:50:45.446 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:50:40 2024", 2024-06-03T04:50:45.446 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:50:45.446 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 2, 2024-06-03T04:50:45.446 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 1, 2024-06-03T04:50:45.446 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "back", 2024-06-03T04:50:45.446 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:50:45.446 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.573, 2024-06-03T04:50:45.446 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.529, 2024-06-03T04:50:45.446 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.524 2024-06-03T04:50:45.446 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.447 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:50:45.447 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.473, 2024-06-03T04:50:45.447 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.463, 2024-06-03T04:50:45.447 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.310 2024-06-03T04:50:45.447 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.447 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:50:45.447 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.630, 2024-06-03T04:50:45.447 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.638, 2024-06-03T04:50:45.447 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.643 2024-06-03T04:50:45.447 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.447 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.237 2024-06-03T04:50:45.447 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.447 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:50:45.447 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:50:40 2024", 2024-06-03T04:50:45.448 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:50:45.448 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 2, 2024-06-03T04:50:45.448 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 0, 2024-06-03T04:50:45.448 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "front", 2024-06-03T04:50:45.448 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:50:45.448 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.561, 2024-06-03T04:50:45.448 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.523, 2024-06-03T04:50:45.448 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.520 2024-06-03T04:50:45.448 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.448 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:50:45.448 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.531, 2024-06-03T04:50:45.448 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.403, 2024-06-03T04:50:45.448 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.281 2024-06-03T04:50:45.448 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.449 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:50:45.449 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.603, 2024-06-03T04:50:45.449 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.616, 2024-06-03T04:50:45.449 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.616 2024-06-03T04:50:45.449 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.450 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.252 2024-06-03T04:50:45.450 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.450 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:50:45.450 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:50:43 2024", 2024-06-03T04:50:45.450 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:50:45.450 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 1, 2024-06-03T04:50:45.450 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 0, 2024-06-03T04:50:45.450 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "front", 2024-06-03T04:50:45.450 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:50:45.450 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.513, 2024-06-03T04:50:45.451 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.555, 2024-06-03T04:50:45.451 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.541 2024-06-03T04:50:45.452 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.452 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:50:45.452 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.254, 2024-06-03T04:50:45.452 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.254, 2024-06-03T04:50:45.452 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.254 2024-06-03T04:50:45.452 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.452 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:50:45.452 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.639, 2024-06-03T04:50:45.452 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.694, 2024-06-03T04:50:45.452 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.694 2024-06-03T04:50:45.452 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.452 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.254 2024-06-03T04:50:45.452 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.453 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:50:45.453 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:50:42 2024", 2024-06-03T04:50:45.453 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:50:45.453 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 0, 2024-06-03T04:50:45.453 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 1, 2024-06-03T04:50:45.453 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "back", 2024-06-03T04:50:45.453 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:50:45.453 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.487, 2024-06-03T04:50:45.454 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.539, 2024-06-03T04:50:45.454 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.508 2024-06-03T04:50:45.454 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.454 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:50:45.454 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.442, 2024-06-03T04:50:45.454 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.429, 2024-06-03T04:50:45.454 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.280 2024-06-03T04:50:45.454 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.454 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:50:45.455 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.535, 2024-06-03T04:50:45.455 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.637, 2024-06-03T04:50:45.455 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.649 2024-06-03T04:50:45.455 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.455 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.466 2024-06-03T04:50:45.455 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.455 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:50:45.455 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:50:42 2024", 2024-06-03T04:50:45.455 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:50:45.455 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 0, 2024-06-03T04:50:45.455 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 1, 2024-06-03T04:50:45.455 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "front", 2024-06-03T04:50:45.455 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:50:45.455 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.534, 2024-06-03T04:50:45.456 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.517, 2024-06-03T04:50:45.456 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.517 2024-06-03T04:50:45.456 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.456 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:50:45.456 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.457, 2024-06-03T04:50:45.456 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.450, 2024-06-03T04:50:45.456 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.222 2024-06-03T04:50:45.456 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.456 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:50:45.456 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.608, 2024-06-03T04:50:45.456 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.608, 2024-06-03T04:50:45.456 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.699 2024-06-03T04:50:45.456 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.456 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.509 2024-06-03T04:50:45.456 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.457 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:50:45.457 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:50:41 2024", 2024-06-03T04:50:45.457 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:50:45.457 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 0, 2024-06-03T04:50:45.457 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 2, 2024-06-03T04:50:45.457 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "back", 2024-06-03T04:50:45.457 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:50:45.457 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.528, 2024-06-03T04:50:45.457 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.522, 2024-06-03T04:50:45.457 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.530 2024-06-03T04:50:45.457 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.457 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:50:45.457 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.460, 2024-06-03T04:50:45.457 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.452, 2024-06-03T04:50:45.458 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.254 2024-06-03T04:50:45.458 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.458 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:50:45.458 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.660, 2024-06-03T04:50:45.458 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.660, 2024-06-03T04:50:45.458 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.683 2024-06-03T04:50:45.458 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.458 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.423 2024-06-03T04:50:45.458 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.458 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:50:45.458 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:50:40 2024", 2024-06-03T04:50:45.458 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:50:45.458 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 2, 2024-06-03T04:50:45.459 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 0, 2024-06-03T04:50:45.459 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "back", 2024-06-03T04:50:45.459 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:50:45.459 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.482, 2024-06-03T04:50:45.459 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.486, 2024-06-03T04:50:45.459 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.499 2024-06-03T04:50:45.459 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.459 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:50:45.459 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.453, 2024-06-03T04:50:45.459 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.398, 2024-06-03T04:50:45.459 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.303 2024-06-03T04:50:45.459 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.459 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:50:45.459 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.511, 2024-06-03T04:50:45.460 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.645, 2024-06-03T04:50:45.460 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.687 2024-06-03T04:50:45.460 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.460 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.215 2024-06-03T04:50:45.460 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.460 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T04:50:45.460 INFO:tasks.workunit.client.0.smithi088.stdout: "last update": "Mon Jun 3 04:50:43 2024", 2024-06-03T04:50:45.460 INFO:tasks.workunit.client.0.smithi088.stdout: "stale": false, 2024-06-03T04:50:45.460 INFO:tasks.workunit.client.0.smithi088.stdout: "from osd": 1, 2024-06-03T04:50:45.460 INFO:tasks.workunit.client.0.smithi088.stdout: "to osd": 0, 2024-06-03T04:50:45.460 INFO:tasks.workunit.client.0.smithi088.stdout: "interface": "back", 2024-06-03T04:50:45.460 INFO:tasks.workunit.client.0.smithi088.stdout: "average": { 2024-06-03T04:50:45.460 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.423, 2024-06-03T04:50:45.460 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.480, 2024-06-03T04:50:45.460 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.490 2024-06-03T04:50:45.461 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.461 INFO:tasks.workunit.client.0.smithi088.stdout: "min": { 2024-06-03T04:50:45.461 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.227, 2024-06-03T04:50:45.461 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.227, 2024-06-03T04:50:45.461 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.227 2024-06-03T04:50:45.461 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.461 INFO:tasks.workunit.client.0.smithi088.stdout: "max": { 2024-06-03T04:50:45.461 INFO:tasks.workunit.client.0.smithi088.stdout: "1min": 0.530, 2024-06-03T04:50:45.461 INFO:tasks.workunit.client.0.smithi088.stdout: "5min": 0.568, 2024-06-03T04:50:45.461 INFO:tasks.workunit.client.0.smithi088.stdout: "15min": 0.615 2024-06-03T04:50:45.461 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T04:50:45.461 INFO:tasks.workunit.client.0.smithi088.stdout: "last": 0.227 2024-06-03T04:50:45.461 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T04:50:45.461 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-06-03T04:50:45.461 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-06-03T04:50:45.462 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: cat td/network-ping/json 2024-06-03T04:50:45.462 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: jq '.entries | length' 2024-06-03T04:50:45.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: test 12 = 12 2024-06-03T04:50:45.490 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: cat td/network-ping/json 2024-06-03T04:50:45.490 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: jq .threshold 2024-06-03T04:50:45.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: test 0 = 0 2024-06-03T04:50:45.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: tee td/network-ping/json 2024-06-03T04:50:45.529 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: get_asok_path mgr.x 2024-06-03T04:50:45.529 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-06-03T04:50:45.529 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-06-03T04:50:45.530 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:50:45.530 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:50:45.530 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34744 2024-06-03T04:50:45.530 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34744/ceph-mgr.x.asok 2024-06-03T04:50:45.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: CEPH_ARGS= 2024-06-03T04:50:45.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34744/ceph-mgr.x.asok dump_osd_network 99 2024-06-03T04:50:45.620 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-06-03T04:50:45.620 INFO:tasks.workunit.client.0.smithi088.stdout: "threshold": 99, 2024-06-03T04:50:45.620 INFO:tasks.workunit.client.0.smithi088.stdout: "entries": [] 2024-06-03T04:50:45.620 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-06-03T04:50:45.634 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: cat td/network-ping/json 2024-06-03T04:50:45.635 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: jq .threshold 2024-06-03T04:50:45.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: test 99 = 99 2024-06-03T04:50:45.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: tee td/network-ping/json 2024-06-03T04:50:45.667 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: get_asok_path osd.0 2024-06-03T04:50:45.667 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T04:50:45.667 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T04:50:45.668 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:50:45.668 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:50:45.668 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34744 2024-06-03T04:50:45.668 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34744/ceph-osd.0.asok 2024-06-03T04:50:45.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: CEPH_ARGS= 2024-06-03T04:50:45.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34744/ceph-osd.0.asok dump_osd_network 98 2024-06-03T04:50:45.762 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-06-03T04:50:45.763 INFO:tasks.workunit.client.0.smithi088.stdout: "threshold": 98, 2024-06-03T04:50:45.763 INFO:tasks.workunit.client.0.smithi088.stdout: "entries": [] 2024-06-03T04:50:45.763 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-06-03T04:50:45.776 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: cat td/network-ping/json 2024-06-03T04:50:45.776 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: jq .threshold 2024-06-03T04:50:45.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: test 98 = 98 2024-06-03T04:50:45.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:105: TEST_network_ping_test1: rm -f td/network-ping/json 2024-06-03T04:50:45.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:20: run: teardown td/network-ping 2024-06-03T04:50:45.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-06-03T04:50:45.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T04:50:45.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-06-03T04:50:45.812 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:50:45.812 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:50:45.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:50:45.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:50:45.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:50:46.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:50:46.124 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:50:46.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:50:46.125 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:50:46.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:50:46.127 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:50:46.127 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:50:46.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:50:46.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:50:46.129 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:50:46.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:50:46.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:50:46.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T04:50:46.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-06-03T04:50:46.156 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:50:46.157 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:50:46.157 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34744 2024-06-03T04:50:46.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.34744 2024-06-03T04:50:46.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:50:46.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:50:46.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:17: run: for func in $funcs 2024-06-03T04:50:46.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:18: run: setup td/network-ping 2024-06-03T04:50:46.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/network-ping 2024-06-03T04:50:46.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/network-ping 2024-06-03T04:50:46.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-06-03T04:50:46.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T04:50:46.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-06-03T04:50:46.160 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:50:46.160 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:50:46.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:50:46.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:50:46.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:50:46.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:50:46.163 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:50:46.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:50:46.164 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:50:46.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:50:46.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:50:46.166 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:50:46.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:50:46.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:50:46.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:50:46.167 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:50:46.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:50:46.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T04:50:46.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-06-03T04:50:46.171 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:50:46.171 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:50:46.172 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34744 2024-06-03T04:50:46.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.34744 2024-06-03T04:50:46.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:50:46.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:50:46.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/network-ping 2024-06-03T04:50:46.174 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T04:50:46.174 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:50:46.174 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34744 2024-06-03T04:50:46.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.34744 2024-06-03T04:50:46.178 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T04:50:46.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T04:50:46.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T04:50:46.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/network-ping 1' TERM HUP INT 2024-06-03T04:50:46.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:19: run: TEST_network_ping_test2 td/network-ping 2024-06-03T04:50:46.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:111: TEST_network_ping_test2: local dir=td/network-ping 2024-06-03T04:50:46.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:113: TEST_network_ping_test2: export CEPH_ARGS 2024-06-03T04:50:46.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:114: TEST_network_ping_test2: export 'EXTRA_OPTS= --mon_warn_on_slow_ping_time=0.001' 2024-06-03T04:50:46.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:114: TEST_network_ping_test2: EXTRA_OPTS=' --mon_warn_on_slow_ping_time=0.001' 2024-06-03T04:50:46.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:115: TEST_network_ping_test2: run_mon td/network-ping a 2024-06-03T04:50:46.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/network-ping 2024-06-03T04:50:46.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T04:50:46.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T04:50:46.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T04:50:46.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/network-ping/a 2024-06-03T04:50:46.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/network-ping/a --run-dir=td/network-ping 2024-06-03T04:50:46.590 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T04:50:46.591 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:50:46.591 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:50:46.591 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:50:46.591 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:50:46.591 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34744 2024-06-03T04:50:46.592 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34744/$cluster-$name.asok' 2024-06-03T04:50:46.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --mon_warn_on_slow_ping_time=0.001 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/network-ping/a '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.34744/$cluster-$name.asok' --mon-cluster-log-file=td/network-ping/log --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-03T04:50:46.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T04:50:46.826 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T04:50:46.826 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:50:46.826 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T04:50:46.826 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T04:50:46.826 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T04:50:46.827 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T04:50:46.827 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T04:50:46.827 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T04:50:46.828 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:50:46.828 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:50:46.828 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34744 2024-06-03T04:50:46.828 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34744/ceph-mon.a.asok 2024-06-03T04:50:46.828 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:50:46.828 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.34744/ceph-mon.a.asok config get fsid 2024-06-03T04:50:46.944 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T04:50:46.944 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:50:46.944 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T04:50:46.944 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T04:50:46.945 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T04:50:46.945 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T04:50:46.945 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T04:50:46.945 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T04:50:46.945 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:50:46.946 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:50:46.946 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34744 2024-06-03T04:50:46.946 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34744/ceph-mon.a.asok 2024-06-03T04:50:46.946 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:50:46.946 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.34744/ceph-mon.a.asok config get mon_host 2024-06-03T04:50:47.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:116: TEST_network_ping_test2: run_mgr td/network-ping x 2024-06-03T04:50:47.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/network-ping 2024-06-03T04:50:47.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T04:50:47.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T04:50:47.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T04:50:47.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/network-ping/x 2024-06-03T04:50:47.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T04:50:47.981 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T04:50:47.981 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:50:47.981 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:50:47.981 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:50:47.981 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:50:47.981 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34744 2024-06-03T04:50:47.981 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34744/$cluster-$name.asok' 2024-06-03T04:50:47.982 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T04:50:47.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --mon_warn_on_slow_ping_time=0.001 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/network-ping/x '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.34744/$cluster-$name.asok' --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T04:50:48.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:117: TEST_network_ping_test2: run_osd td/network-ping 0 2024-06-03T04:50:48.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-06-03T04:50:48.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T04:50:48.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T04:50:48.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T04:50:48.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/0 2024-06-03T04:50:48.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9ee5db8b-6e0e-4fdb-a603-cfc7f9259256 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-06-03T04:50:48.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T04:50:48.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T04:50:48.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T04:50:48.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/0' 2024-06-03T04:50:48.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/0/journal' 2024-06-03T04:50:48.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T04:50:48.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2024-06-03T04:50:48.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-06-03T04:50:48.021 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T04:50:48.021 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:50:48.021 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:50:48.021 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:50:48.021 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:50:48.022 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34744 2024-06-03T04:50:48.022 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34744/$cluster-$name.asok' 2024-06-03T04:50:48.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34744/$cluster-$name.asok' 2024-06-03T04:50:48.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T04:50:48.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T04:50:48.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T04:50:48.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-06-03T04:50:48.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-06-03T04:50:48.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T04:50:48.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T04:50:48.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T04:50:48.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T04:50:48.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T04:50:48.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T04:50:48.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/0 2024-06-03T04:50:48.023 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T04:50:48.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=62558622-4132-4e88-a17b-df309a27bc81 2024-06-03T04:50:48.023 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 62558622-4132-4e88-a17b-df309a27bc81 2024-06-03T04:50:48.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 62558622-4132-4e88-a17b-df309a27bc81' 2024-06-03T04:50:48.024 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T04:50:48.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCoS11m8hEtAhAANwwZRU4zg1vqXLrv0NMWRg== 2024-06-03T04:50:48.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCoS11m8hEtAhAANwwZRU4zg1vqXLrv0NMWRg=="}' 2024-06-03T04:50:48.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 62558622-4132-4e88-a17b-df309a27bc81 -i td/network-ping/0/new.json 2024-06-03T04:50:48.373 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T04:50:48.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/0/new.json 2024-06-03T04:50:48.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=9ee5db8b-6e0e-4fdb-a603-cfc7f9259256 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.34744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCoS11m8hEtAhAANwwZRU4zg1vqXLrv0NMWRg== --osd-uuid 62558622-4132-4e88-a17b-df309a27bc81 2024-06-03T04:50:48.420 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:50:48.415+0000 7fdda01f17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:50:48.450 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:50:48.443+0000 7fdda01f17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:50:48.451 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:50:48.447+0000 7fdda01f17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:50:48.451 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:50:48.447+0000 7fdda01f17c0 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-06-03T04:50:48.451 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:50:48.447+0000 7fdda01f17c0 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-06-03T04:50:48.451 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:50:48.447+0000 7fdda01f17c0 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-06-03T04:50:48.498 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:50:48.491+0000 7fdda01f17c0 -1 bluestore(td/network-ping/0) _read_fsid unparsable uuid 2024-06-03T04:50:54.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/0/keyring 2024-06-03T04:50:54.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T04:50:54.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T04:50:54.879 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T04:50:54.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T04:50:55.432 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T04:50:55.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T04:50:55.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=9ee5db8b-6e0e-4fdb-a603-cfc7f9259256 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.34744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T04:50:55.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T04:50:55.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T04:50:55.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T04:50:55.466 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:50:55.459+0000 7f1a915437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:50:55.468 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:50:55.463+0000 7f1a915437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:50:55.469 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:50:55.463+0000 7f1a915437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:50:55.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T04:50:55.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T04:50:55.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T04:50:55.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T04:50:55.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T04:50:55.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:50:55.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T04:50:55.884 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T04:50:55.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:50:55.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:50:56.044 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:50:56.039+0000 7f1a915437c0 -1 Falling back to public interface 2024-06-03T04:50:56.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:50:57.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:50:57.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:50:57.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T04:50:57.291 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T04:50:57.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:50:57.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:50:57.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:50:58.640 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:50:58.635+0000 7f1a915437c0 -1 osd.0 0 log_to_monitors true 2024-06-03T04:50:58.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:50:58.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:50:58.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T04:50:58.698 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T04:50:58.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:50:58.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:50:59.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:51:00.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:51:00.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:51:00.383 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T04:51:00.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T04:51:00.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:51:00.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:51:00.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:51:01.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:51:01.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:51:01.822 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T04:51:01.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T04:51:01.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:51:01.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:51:02.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:51:03.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:51:03.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:51:03.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T04:51:03.244 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T04:51:03.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:51:03.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:51:03.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:51:04.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:51:04.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:51:04.956 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T04:51:04.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T04:51:04.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:51:04.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:51:05.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:51:06.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:51:06.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:51:06.534 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T04:51:06.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T04:51:06.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:51:06.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:51:06.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:51:07.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:51:07.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:51:07.961 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T04:51:07.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T04:51:07.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:51:07.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:51:08.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:51:09.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:51:09.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:51:09.531 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T04:51:09.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T04:51:09.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:51:09.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:51:09.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:51:10.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:51:10.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:51:10.945 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T04:51:10.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T04:51:10.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:51:10.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:51:11.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:51:12.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:51:12.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:51:12.391 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T04:51:12.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T04:51:12.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:51:12.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:51:12.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:51:13.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:51:13.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:51:13.853 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T04:51:13.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T04:51:13.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:51:13.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:51:14.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:51:15.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:51:15.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:51:15.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T04:51:15.676 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T04:51:15.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:51:15.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:51:16.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:51:17.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:51:17.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:51:17.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T04:51:17.913 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T04:51:17.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:51:17.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:51:18.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:51:19.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:51:19.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:51:19.862 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T04:51:19.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T04:51:19.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:51:19.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:51:20.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:51:21.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:51:21.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:51:21.630 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T04:51:21.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T04:51:21.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:51:21.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:51:22.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:51:23.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:51:23.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:51:23.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T04:51:23.292 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T04:51:23.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:51:23.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:51:23.741 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2086227954,v1:127.0.0.1:6803/2086227954] [v2:127.0.0.1:6804/2086227954,v1:127.0.0.1:6805/2086227954] exists,up 62558622-4132-4e88-a17b-df309a27bc81 2024-06-03T04:51:23.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T04:51:23.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T04:51:23.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T04:51:23.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:118: TEST_network_ping_test2: run_osd td/network-ping 1 2024-06-03T04:51:23.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-06-03T04:51:23.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T04:51:23.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-03T04:51:23.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T04:51:23.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/1 2024-06-03T04:51:23.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9ee5db8b-6e0e-4fdb-a603-cfc7f9259256 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-06-03T04:51:23.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T04:51:23.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T04:51:23.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T04:51:23.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/1' 2024-06-03T04:51:23.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/1/journal' 2024-06-03T04:51:23.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T04:51:23.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2024-06-03T04:51:23.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-06-03T04:51:23.748 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T04:51:23.748 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:51:23.748 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:51:23.749 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:51:23.749 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:51:23.749 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34744 2024-06-03T04:51:23.749 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34744/$cluster-$name.asok' 2024-06-03T04:51:23.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34744/$cluster-$name.asok' 2024-06-03T04:51:23.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T04:51:23.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T04:51:23.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T04:51:23.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-06-03T04:51:23.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-06-03T04:51:23.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T04:51:23.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T04:51:23.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T04:51:23.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T04:51:23.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T04:51:23.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T04:51:23.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/1 2024-06-03T04:51:23.750 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T04:51:23.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9816e414-8be1-456e-97eb-0577de840e43 2024-06-03T04:51:23.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 9816e414-8be1-456e-97eb-0577de840e43' 2024-06-03T04:51:23.751 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 9816e414-8be1-456e-97eb-0577de840e43 2024-06-03T04:51:23.751 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T04:51:23.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDLS11mJ8LALRAAFHmMroGkovkWT6jn7T+OOg== 2024-06-03T04:51:23.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDLS11mJ8LALRAAFHmMroGkovkWT6jn7T+OOg=="}' 2024-06-03T04:51:23.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9816e414-8be1-456e-97eb-0577de840e43 -i td/network-ping/1/new.json 2024-06-03T04:51:24.262 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T04:51:24.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/1/new.json 2024-06-03T04:51:24.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=9ee5db8b-6e0e-4fdb-a603-cfc7f9259256 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.34744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDLS11mJ8LALRAAFHmMroGkovkWT6jn7T+OOg== --osd-uuid 9816e414-8be1-456e-97eb-0577de840e43 2024-06-03T04:51:24.319 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:51:24.311+0000 7f9a5726f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:51:24.340 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:51:24.335+0000 7f9a5726f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:51:24.341 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:51:24.335+0000 7f9a5726f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:51:24.341 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:51:24.335+0000 7f9a5726f7c0 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-06-03T04:51:24.341 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:51:24.335+0000 7f9a5726f7c0 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-06-03T04:51:24.342 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:51:24.335+0000 7f9a5726f7c0 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-06-03T04:51:24.386 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:51:24.379+0000 7f9a5726f7c0 -1 bluestore(td/network-ping/1) _read_fsid unparsable uuid 2024-06-03T04:51:31.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/1/keyring 2024-06-03T04:51:31.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T04:51:31.011 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-06-03T04:51:31.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-03T04:51:31.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T04:51:31.641 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-06-03T04:51:31.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-03T04:51:31.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T04:51:31.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T04:51:31.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T04:51:31.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=9ee5db8b-6e0e-4fdb-a603-cfc7f9259256 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.34744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T04:51:31.675 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:51:31.671+0000 7f19bef537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:51:31.695 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:51:31.687+0000 7f19bef537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:51:31.696 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:51:31.691+0000 7f19bef537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:51:32.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-03T04:51:32.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T04:51:32.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T04:51:32.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T04:51:32.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T04:51:32.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:51:32.075 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T04:51:32.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T04:51:32.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:51:32.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:51:32.261 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:51:32.255+0000 7f19bef537c0 -1 Falling back to public interface 2024-06-03T04:51:32.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:51:33.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:51:33.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:51:33.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T04:51:33.479 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T04:51:33.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:51:33.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:51:33.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:51:34.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:51:34.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:51:34.872 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T04:51:34.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T04:51:34.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:51:34.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:51:34.925 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:51:34.919+0000 7f19bef537c0 -1 osd.1 0 log_to_monitors true 2024-06-03T04:51:35.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:51:36.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:51:36.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:51:36.460 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T04:51:36.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T04:51:36.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:51:36.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:51:37.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:51:38.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:51:38.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:51:38.022 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T04:51:38.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T04:51:38.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:51:38.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:51:38.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:51:39.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:51:39.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:51:39.487 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T04:51:39.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T04:51:39.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:51:39.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:51:39.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:51:40.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:51:40.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:51:40.968 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T04:51:40.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T04:51:40.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:51:40.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:51:41.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:51:42.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:51:42.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:51:42.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T04:51:42.542 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T04:51:42.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:51:42.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:51:43.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:51:44.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:51:44.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:51:44.162 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T04:51:44.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T04:51:44.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:51:44.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:51:44.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:51:45.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:51:45.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:51:45.608 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T04:51:45.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T04:51:45.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:51:45.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:51:46.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:51:47.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:51:47.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:51:47.078 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T04:51:47.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T04:51:47.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:51:47.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:51:47.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:51:48.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:51:48.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:51:48.601 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T04:51:48.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T04:51:48.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:51:48.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:51:49.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:51:50.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:51:50.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:51:50.237 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T04:51:50.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T04:51:50.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:51:50.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:51:51.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:51:52.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:51:52.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:51:52.329 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T04:51:52.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T04:51:52.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:51:52.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:51:53.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:51:54.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:51:54.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:51:54.354 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T04:51:54.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T04:51:54.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:51:54.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:51:55.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:51:56.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:51:56.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:51:56.143 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T04:51:56.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T04:51:56.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:51:56.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:51:57.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:51:58.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:51:58.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:51:58.068 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T04:51:58.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T04:51:58.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:51:58.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:51:59.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:52:00.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:52:00.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:52:00.179 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T04:52:00.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T04:52:00.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:52:00.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:52:00.700 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1898875988,v1:127.0.0.1:6811/1898875988] [v2:127.0.0.1:6812/1898875988,v1:127.0.0.1:6813/1898875988] exists,up 9816e414-8be1-456e-97eb-0577de840e43 2024-06-03T04:52:00.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T04:52:00.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T04:52:00.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T04:52:00.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:119: TEST_network_ping_test2: run_osd td/network-ping 2 2024-06-03T04:52:00.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-06-03T04:52:00.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T04:52:00.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-03T04:52:00.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T04:52:00.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/2 2024-06-03T04:52:00.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9ee5db8b-6e0e-4fdb-a603-cfc7f9259256 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-06-03T04:52:00.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T04:52:00.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T04:52:00.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T04:52:00.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/2' 2024-06-03T04:52:00.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/2/journal' 2024-06-03T04:52:00.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T04:52:00.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2024-06-03T04:52:00.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-06-03T04:52:00.705 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T04:52:00.705 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:52:00.705 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:52:00.705 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:52:00.705 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:52:00.705 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34744 2024-06-03T04:52:00.705 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34744/$cluster-$name.asok' 2024-06-03T04:52:00.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34744/$cluster-$name.asok' 2024-06-03T04:52:00.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T04:52:00.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T04:52:00.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T04:52:00.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-06-03T04:52:00.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-06-03T04:52:00.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T04:52:00.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T04:52:00.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T04:52:00.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T04:52:00.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T04:52:00.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T04:52:00.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/2 2024-06-03T04:52:00.707 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T04:52:00.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3c24de20-efc4-4356-9401-1e4ec298e67d 2024-06-03T04:52:00.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 3c24de20-efc4-4356-9401-1e4ec298e67d' 2024-06-03T04:52:00.707 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 3c24de20-efc4-4356-9401-1e4ec298e67d 2024-06-03T04:52:00.707 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T04:52:00.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDwS11m+V8rKxAAjbqs6ActdK2JI439H6ZX7w== 2024-06-03T04:52:00.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDwS11m+V8rKxAAjbqs6ActdK2JI439H6ZX7w=="}' 2024-06-03T04:52:00.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3c24de20-efc4-4356-9401-1e4ec298e67d -i td/network-ping/2/new.json 2024-06-03T04:52:01.338 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T04:52:01.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/2/new.json 2024-06-03T04:52:01.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=9ee5db8b-6e0e-4fdb-a603-cfc7f9259256 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.34744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDwS11m+V8rKxAAjbqs6ActdK2JI439H6ZX7w== --osd-uuid 3c24de20-efc4-4356-9401-1e4ec298e67d 2024-06-03T04:52:01.391 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:52:01.383+0000 7f9e60c667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:52:01.433 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:52:01.427+0000 7f9e60c667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:52:01.434 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:52:01.427+0000 7f9e60c667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:52:01.435 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:52:01.427+0000 7f9e60c667c0 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-06-03T04:52:01.435 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:52:01.427+0000 7f9e60c667c0 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-06-03T04:52:01.435 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:52:01.427+0000 7f9e60c667c0 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-06-03T04:52:01.470 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:52:01.463+0000 7f9e60c667c0 -1 bluestore(td/network-ping/2) _read_fsid unparsable uuid 2024-06-03T04:52:08.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/2/keyring 2024-06-03T04:52:08.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T04:52:08.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-03T04:52:08.280 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-06-03T04:52:08.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T04:52:08.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-03T04:52:08.865 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-06-03T04:52:08.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T04:52:08.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=9ee5db8b-6e0e-4fdb-a603-cfc7f9259256 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.34744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T04:52:08.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T04:52:08.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T04:52:08.898 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:52:08.891+0000 7fab20f417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:52:08.900 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:52:08.895+0000 7fab20f417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:52:08.901 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:52:08.895+0000 7fab20f417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:52:09.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-03T04:52:09.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T04:52:09.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T04:52:09.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T04:52:09.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T04:52:09.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:52:09.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T04:52:09.295 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T04:52:09.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:52:09.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:52:09.444 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:52:09.439+0000 7fab20f417c0 -1 Falling back to public interface 2024-06-03T04:52:09.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:52:10.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:52:10.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:52:10.735 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T04:52:10.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T04:52:10.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:52:10.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:52:11.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:52:12.106 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:52:12.099+0000 7fab20f417c0 -1 osd.2 0 log_to_monitors true 2024-06-03T04:52:12.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:52:12.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:52:12.147 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T04:52:12.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T04:52:12.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:52:12.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:52:12.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:52:13.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:52:13.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:52:13.916 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T04:52:13.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T04:52:13.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:52:13.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:52:14.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:52:15.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:52:15.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:52:15.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T04:52:15.360 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T04:52:15.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:52:15.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:52:15.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:52:16.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:52:16.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:52:16.780 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T04:52:16.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T04:52:16.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:52:16.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:52:17.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:52:18.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:52:18.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:52:18.421 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T04:52:18.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T04:52:18.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:52:18.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:52:19.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:52:20.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:52:20.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:52:20.035 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T04:52:20.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T04:52:20.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:52:20.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:52:20.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:52:21.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:52:21.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:52:21.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T04:52:21.492 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T04:52:21.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:52:21.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:52:22.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:52:23.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:52:23.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:52:23.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T04:52:23.094 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T04:52:23.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:52:23.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:52:23.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:52:24.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:52:24.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:52:24.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T04:52:24.610 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T04:52:24.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:52:24.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:52:25.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:52:26.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:52:26.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:52:26.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T04:52:26.101 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T04:52:26.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:52:26.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:52:26.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:52:27.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:52:27.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:52:27.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T04:52:27.825 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T04:52:27.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:52:27.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:52:28.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:52:29.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:52:29.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:52:29.619 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T04:52:29.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T04:52:29.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:52:29.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:52:30.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:52:31.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:52:31.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:52:31.502 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T04:52:31.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T04:52:31.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:52:31.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:52:32.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:52:33.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:52:33.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:52:33.452 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T04:52:33.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T04:52:33.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:52:33.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:52:34.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:52:35.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:52:35.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:52:35.498 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T04:52:35.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T04:52:35.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:52:35.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:52:36.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:52:37.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:52:37.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:52:37.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T04:52:37.544 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T04:52:37.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:52:37.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:52:37.994 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3463689113,v1:127.0.0.1:6819/3463689113] [v2:127.0.0.1:6820/3463689113,v1:127.0.0.1:6821/3463689113] exists,up 3c24de20-efc4-4356-9401-1e4ec298e67d 2024-06-03T04:52:37.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T04:52:37.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T04:52:37.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T04:52:37.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:121: TEST_network_ping_test2: sleep 5 2024-06-03T04:52:42.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:122: TEST_network_ping_test2: ceph osd crush add-bucket dc1 datacenter 2024-06-03T04:52:43.536 INFO:tasks.workunit.client.0.smithi088.stderr:bucket 'dc1' already exists 2024-06-03T04:52:43.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:123: TEST_network_ping_test2: ceph osd crush add-bucket dc2 datacenter 2024-06-03T04:52:44.215 INFO:tasks.workunit.client.0.smithi088.stderr:bucket 'dc2' already exists 2024-06-03T04:52:44.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:124: TEST_network_ping_test2: ceph osd crush add-bucket dc3 datacenter 2024-06-03T04:52:44.888 INFO:tasks.workunit.client.0.smithi088.stderr:bucket 'dc3' already exists 2024-06-03T04:52:44.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:125: TEST_network_ping_test2: ceph osd crush add-bucket rack1 rack 2024-06-03T04:52:45.454 INFO:tasks.workunit.client.0.smithi088.stderr:bucket 'rack1' already exists 2024-06-03T04:52:45.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:126: TEST_network_ping_test2: ceph osd crush add-bucket rack2 rack 2024-06-03T04:52:45.994 INFO:tasks.workunit.client.0.smithi088.stderr:bucket 'rack2' already exists 2024-06-03T04:52:46.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:127: TEST_network_ping_test2: ceph osd crush add-bucket rack3 rack 2024-06-03T04:52:46.637 INFO:tasks.workunit.client.0.smithi088.stderr:bucket 'rack3' already exists 2024-06-03T04:52:46.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:128: TEST_network_ping_test2: ceph osd crush add-bucket host1 host 2024-06-03T04:52:47.398 INFO:tasks.workunit.client.0.smithi088.stderr:bucket 'host1' already exists 2024-06-03T04:52:47.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:129: TEST_network_ping_test2: ceph osd crush add-bucket host2 host 2024-06-03T04:52:47.959 INFO:tasks.workunit.client.0.smithi088.stderr:bucket 'host2' already exists 2024-06-03T04:52:47.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:130: TEST_network_ping_test2: ceph osd crush add-bucket host3 host 2024-06-03T04:52:48.629 INFO:tasks.workunit.client.0.smithi088.stderr:bucket 'host3' already exists 2024-06-03T04:52:48.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:131: TEST_network_ping_test2: ceph osd crush move dc1 root=default 2024-06-03T04:52:49.433 INFO:tasks.workunit.client.0.smithi088.stderr:no need to move item id -5 name 'dc1' to location {root=default} in crush map 2024-06-03T04:52:49.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:132: TEST_network_ping_test2: ceph osd crush move dc2 root=default 2024-06-03T04:52:50.036 INFO:tasks.workunit.client.0.smithi088.stderr:no need to move item id -6 name 'dc2' to location {root=default} in crush map 2024-06-03T04:52:50.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:133: TEST_network_ping_test2: ceph osd crush move dc3 root=default 2024-06-03T04:52:50.655 INFO:tasks.workunit.client.0.smithi088.stderr:no need to move item id -7 name 'dc3' to location {root=default} in crush map 2024-06-03T04:52:50.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:134: TEST_network_ping_test2: ceph osd crush move rack1 datacenter=dc1 2024-06-03T04:52:51.242 INFO:tasks.workunit.client.0.smithi088.stderr:no need to move item id -8 name 'rack1' to location {datacenter=dc1} in crush map 2024-06-03T04:52:51.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:135: TEST_network_ping_test2: ceph osd crush move rack2 datacenter=dc2 2024-06-03T04:52:51.903 INFO:tasks.workunit.client.0.smithi088.stderr:no need to move item id -9 name 'rack2' to location {datacenter=dc2} in crush map 2024-06-03T04:52:51.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:136: TEST_network_ping_test2: ceph osd crush move rack3 datacenter=dc3 2024-06-03T04:52:52.481 INFO:tasks.workunit.client.0.smithi088.stderr:no need to move item id -10 name 'rack3' to location {datacenter=dc3} in crush map 2024-06-03T04:52:52.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:137: TEST_network_ping_test2: ceph osd crush move host1 rack=rack1 2024-06-03T04:52:53.205 INFO:tasks.workunit.client.0.smithi088.stderr:no need to move item id -11 name 'host1' to location {rack=rack1} in crush map 2024-06-03T04:52:53.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:138: TEST_network_ping_test2: ceph osd crush move host2 rack=rack2 2024-06-03T04:52:53.878 INFO:tasks.workunit.client.0.smithi088.stderr:no need to move item id -12 name 'host2' to location {rack=rack2} in crush map 2024-06-03T04:52:53.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:139: TEST_network_ping_test2: ceph osd crush move host3 rack=rack3 2024-06-03T04:52:54.409 INFO:tasks.workunit.client.0.smithi088.stderr:no need to move item id -13 name 'host3' to location {rack=rack3} in crush map 2024-06-03T04:52:54.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:140: TEST_network_ping_test2: ceph osd crush set osd.0 1.0 host=host1 2024-06-03T04:52:55.118 INFO:tasks.workunit.client.0.smithi088.stderr:set item id 0 name 'osd.0' weight 1 at location {host=host1}: no change 2024-06-03T04:52:55.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:141: TEST_network_ping_test2: ceph osd crush set osd.1 1.0 host=host2 2024-06-03T04:52:55.669 INFO:tasks.workunit.client.0.smithi088.stderr:set item id 1 name 'osd.1' weight 1 at location {host=host2}: no change 2024-06-03T04:52:55.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:142: TEST_network_ping_test2: ceph osd crush set osd.2 1.0 host=host3 2024-06-03T04:52:56.357 INFO:tasks.workunit.client.0.smithi088.stderr:set item id 2 name 'osd.2' weight 1 at location {host=host3}: no change 2024-06-03T04:52:56.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:143: TEST_network_ping_test2: ceph osd crush rule create-simple myrule default host firstn 2024-06-03T04:52:56.934 INFO:tasks.workunit.client.0.smithi088.stderr:rule myrule already exists 2024-06-03T04:52:56.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:145: TEST_network_ping_test2: create_pool foo 16 16 replicated myrule 2024-06-03T04:52:56.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 16 replicated myrule 2024-06-03T04:52:57.487 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'foo' already exists 2024-06-03T04:52:57.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T04:52:58.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:148: TEST_network_ping_test2: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2024-06-03T04:52:58.737 INFO:tasks.workunit.client.0.smithi088.stdout:hints = 1 2024-06-03T04:52:58.738 INFO:tasks.workunit.client.0.smithi088.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2024-06-03T04:52:58.738 INFO:tasks.workunit.client.0.smithi088.stdout:Object prefix: benchmark_data_smithi088_47030 2024-06-03T04:52:58.738 INFO:tasks.workunit.client.0.smithi088.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-03T04:52:58.738 INFO:tasks.workunit.client.0.smithi088.stdout: 0 0 0 0 0 0 - 0 2024-06-03T04:52:59.738 INFO:tasks.workunit.client.0.smithi088.stdout: 1 16 51 35 0.136711 0.136719 0.275323 0.428583 2024-06-03T04:53:00.738 INFO:tasks.workunit.client.0.smithi088.stdout: 2 16 101 85 0.165997 0.195312 0.278845 0.351335 2024-06-03T04:53:01.738 INFO:tasks.workunit.client.0.smithi088.stdout: 3 16 161 145 0.188779 0.234375 0.279846 0.321044 2024-06-03T04:53:02.738 INFO:tasks.workunit.client.0.smithi088.stdout: 4 16 215 199 0.194313 0.210938 0.283621 0.305633 2024-06-03T04:53:03.738 INFO:tasks.workunit.client.0.smithi088.stdout: 5 16 277 261 0.203883 0.242188 0.296214 0.297396 2024-06-03T04:53:04.738 INFO:tasks.workunit.client.0.smithi088.stdout: 6 16 337 321 0.20896 0.234375 0.260661 0.289792 2024-06-03T04:53:05.739 INFO:tasks.workunit.client.0.smithi088.stdout: 7 16 391 375 0.209237 0.210938 0.30039 0.291459 2024-06-03T04:53:06.739 INFO:tasks.workunit.client.0.smithi088.stdout: 8 16 433 417 0.203587 0.164062 0.544792 0.297448 2024-06-03T04:53:07.739 INFO:tasks.workunit.client.0.smithi088.stdout: 9 16 487 471 0.204401 0.210938 0.267469 0.299543 2024-06-03T04:53:08.739 INFO:tasks.workunit.client.0.smithi088.stdout: 10 16 551 535 0.208957 0.25 0.229695 0.295774 2024-06-03T04:53:09.740 INFO:tasks.workunit.client.0.smithi088.stdout:Total time run: 10.1782 2024-06-03T04:53:09.740 INFO:tasks.workunit.client.0.smithi088.stdout:Total writes made: 551 2024-06-03T04:53:09.740 INFO:tasks.workunit.client.0.smithi088.stdout:Write size: 4096 2024-06-03T04:53:09.740 INFO:tasks.workunit.client.0.smithi088.stdout:Object size: 4096 2024-06-03T04:53:09.741 INFO:tasks.workunit.client.0.smithi088.stdout:Bandwidth (MB/sec): 0.211465 2024-06-03T04:53:09.741 INFO:tasks.workunit.client.0.smithi088.stdout:Stddev Bandwidth: 0.0357658 2024-06-03T04:53:09.741 INFO:tasks.workunit.client.0.smithi088.stdout:Max bandwidth (MB/sec): 0.25 2024-06-03T04:53:09.741 INFO:tasks.workunit.client.0.smithi088.stdout:Min bandwidth (MB/sec): 0.136719 2024-06-03T04:53:09.741 INFO:tasks.workunit.client.0.smithi088.stdout:Average IOPS: 54 2024-06-03T04:53:09.741 INFO:tasks.workunit.client.0.smithi088.stdout:Stddev IOPS: 9.15605 2024-06-03T04:53:09.742 INFO:tasks.workunit.client.0.smithi088.stdout:Max IOPS: 64 2024-06-03T04:53:09.742 INFO:tasks.workunit.client.0.smithi088.stdout:Min IOPS: 35 2024-06-03T04:53:09.742 INFO:tasks.workunit.client.0.smithi088.stdout:Average Latency(s): 0.293909 2024-06-03T04:53:09.742 INFO:tasks.workunit.client.0.smithi088.stdout:Stddev Latency(s): 0.0759124 2024-06-03T04:53:09.743 INFO:tasks.workunit.client.0.smithi088.stdout:Max latency(s): 0.695883 2024-06-03T04:53:09.743 INFO:tasks.workunit.client.0.smithi088.stdout:Min latency(s): 0.221482 2024-06-03T04:53:09.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:151: TEST_network_ping_test2: sleep 10 2024-06-03T04:53:19.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:152: TEST_network_ping_test2: flush_pg_stats 2024-06-03T04:53:19.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T04:53:19.833 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T04:53:20.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T04:53:20.261 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-06-03T04:53:20.261 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-06-03T04:53:20.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T04:53:20.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T04:53:20.261 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T04:53:20.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836512 2024-06-03T04:53:20.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836512 2024-06-03T04:53:20.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512' 2024-06-03T04:53:20.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T04:53:20.441 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T04:53:20.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672990 2024-06-03T04:53:20.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672990 2024-06-03T04:53:20.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512 1-42949672990' 2024-06-03T04:53:20.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T04:53:20.593 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T04:53:20.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-06-03T04:53:20.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-06-03T04:53:20.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512 1-42949672990 2-64424509456' 2024-06-03T04:53:20.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T04:53:20.739 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836512 2024-06-03T04:53:20.739 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T04:53:20.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T04:53:20.739 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836512 2024-06-03T04:53:20.739 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T04:53:20.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836512 2024-06-03T04:53:20.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-06-03T04:53:20.740 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836512 2024-06-03T04:53:20.740 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T04:53:21.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836512 -lt 21474836512 2024-06-03T04:53:21.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T04:53:21.121 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672990 2024-06-03T04:53:21.121 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T04:53:21.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T04:53:21.124 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672990 2024-06-03T04:53:21.124 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T04:53:21.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672990 2024-06-03T04:53:21.125 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672990 2024-06-03T04:53:21.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672990' 2024-06-03T04:53:21.126 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T04:53:21.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672989 -lt 42949672990 2024-06-03T04:53:21.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T04:53:22.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T04:53:22.561 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T04:53:22.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672991 -lt 42949672990 2024-06-03T04:53:22.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T04:53:22.966 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T04:53:22.966 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-06-03T04:53:22.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T04:53:22.967 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-06-03T04:53:22.967 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T04:53:22.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-06-03T04:53:22.969 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509456 2024-06-03T04:53:22.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-06-03T04:53:22.969 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T04:53:23.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509456 2024-06-03T04:53:23.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:154: TEST_network_ping_test2: ceph health 2024-06-03T04:53:23.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:154: TEST_network_ping_test2: tee td/network-ping/health 2024-06-03T04:53:23.890 INFO:tasks.workunit.client.0.smithi088.stdout:HEALTH_WARN Slow OSD heartbeats on back (longest 0.583ms); Slow OSD heartbeats on front (longest 0.669ms); 1 pool(s) do not have an application enabled 2024-06-03T04:53:23.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:155: TEST_network_ping_test2: grep -q 'Slow OSD heartbeats' td/network-ping/health 2024-06-03T04:53:23.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:157: TEST_network_ping_test2: ceph health detail 2024-06-03T04:53:23.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:157: TEST_network_ping_test2: tee td/network-ping/health 2024-06-03T04:53:24.467 INFO:tasks.workunit.client.0.smithi088.stdout:HEALTH_WARN Slow OSD heartbeats on back (longest 0.583ms); Slow OSD heartbeats on front (longest 0.669ms); 1 pool(s) do not have an application enabled 2024-06-03T04:53:24.467 INFO:tasks.workunit.client.0.smithi088.stdout:[WRN] OSD_SLOW_PING_TIME_BACK: Slow OSD heartbeats on back (longest 0.583ms) 2024-06-03T04:53:24.467 INFO:tasks.workunit.client.0.smithi088.stdout: Slow OSD heartbeats on back from osd.1 [dc2,rack2] to osd.2 [dc3,rack3] 0.583 msec 2024-06-03T04:53:24.467 INFO:tasks.workunit.client.0.smithi088.stdout: Slow OSD heartbeats on back from osd.0 [dc1,rack1] to osd.2 [dc3,rack3] 0.555 msec 2024-06-03T04:53:24.467 INFO:tasks.workunit.client.0.smithi088.stdout: Slow OSD heartbeats on back from osd.2 [dc3,rack3] to osd.1 [dc2,rack2] 0.537 msec 2024-06-03T04:53:24.467 INFO:tasks.workunit.client.0.smithi088.stdout: Slow OSD heartbeats on back from osd.2 [dc3,rack3] to osd.0 [dc1,rack1] 0.521 msec 2024-06-03T04:53:24.467 INFO:tasks.workunit.client.0.smithi088.stdout: Slow OSD heartbeats on back from osd.1 [dc2,rack2] to osd.0 [dc1,rack1] 0.498 msec possibly improving 2024-06-03T04:53:24.467 INFO:tasks.workunit.client.0.smithi088.stdout: Slow OSD heartbeats on back from osd.0 [dc1,rack1] to osd.1 [dc2,rack2] 0.493 msec 2024-06-03T04:53:24.467 INFO:tasks.workunit.client.0.smithi088.stdout:[WRN] OSD_SLOW_PING_TIME_FRONT: Slow OSD heartbeats on front (longest 0.669ms) 2024-06-03T04:53:24.467 INFO:tasks.workunit.client.0.smithi088.stdout: Slow OSD heartbeats on front from osd.2 [dc3,rack3] to osd.0 [dc1,rack1] 0.669 msec 2024-06-03T04:53:24.467 INFO:tasks.workunit.client.0.smithi088.stdout: Slow OSD heartbeats on front from osd.0 [dc1,rack1] to osd.2 [dc3,rack3] 0.588 msec 2024-06-03T04:53:24.467 INFO:tasks.workunit.client.0.smithi088.stdout: Slow OSD heartbeats on front from osd.2 [dc3,rack3] to osd.1 [dc2,rack2] 0.585 msec 2024-06-03T04:53:24.468 INFO:tasks.workunit.client.0.smithi088.stdout: Slow OSD heartbeats on front from osd.1 [dc2,rack2] to osd.2 [dc3,rack3] 0.576 msec 2024-06-03T04:53:24.468 INFO:tasks.workunit.client.0.smithi088.stdout: Slow OSD heartbeats on front from osd.1 [dc2,rack2] to osd.0 [dc1,rack1] 0.523 msec 2024-06-03T04:53:24.468 INFO:tasks.workunit.client.0.smithi088.stdout: Slow OSD heartbeats on front from osd.0 [dc1,rack1] to osd.1 [dc2,rack2] 0.520 msec 2024-06-03T04:53:24.468 INFO:tasks.workunit.client.0.smithi088.stdout:[WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-06-03T04:53:24.468 INFO:tasks.workunit.client.0.smithi088.stdout: application not enabled on pool 'foo' 2024-06-03T04:53:24.468 INFO:tasks.workunit.client.0.smithi088.stdout: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-06-03T04:53:24.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:158: TEST_network_ping_test2: grep -q OSD_SLOW_PING_TIME_BACK td/network-ping/health 2024-06-03T04:53:24.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:159: TEST_network_ping_test2: grep -q OSD_SLOW_PING_TIME_FRONT td/network-ping/health 2024-06-03T04:53:24.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:160: TEST_network_ping_test2: grep -q 'Slow OSD heartbeats on front from osd[.][0-2] [[]dc[1-3],rack[1-3][]] to osd[.][0-2] [[]dc[1-3],rack[1-3][]]' td/network-ping/health 2024-06-03T04:53:24.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:162: TEST_network_ping_test2: rm -f td/network-ping/health 2024-06-03T04:53:24.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:20: run: teardown td/network-ping 2024-06-03T04:53:24.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-06-03T04:53:24.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T04:53:24.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-06-03T04:53:24.488 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:53:24.488 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:53:24.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:53:24.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:53:24.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:53:24.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:53:24.799 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:53:24.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:53:24.801 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:53:24.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:53:24.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:53:24.803 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:53:24.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:53:24.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:53:24.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:53:24.805 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:53:24.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:53:24.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T04:53:24.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-06-03T04:53:24.832 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:53:24.832 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:53:24.832 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34744 2024-06-03T04:53:24.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.34744 2024-06-03T04:53:24.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:53:24.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:53:24.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-06-03T04:53:24.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/network-ping 0 2024-06-03T04:53:24.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-06-03T04:53:24.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-06-03T04:53:24.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-06-03T04:53:24.835 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:53:24.835 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:53:24.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:53:24.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:53:24.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:53:24.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:53:24.837 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:53:24.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:53:24.838 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:53:24.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:53:24.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:53:24.840 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:53:24.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:53:24.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:53:24.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:53:24.841 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:53:24.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:53:24.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-06-03T04:53:24.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-06-03T04:53:24.845 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:53:24.845 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:53:24.845 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34744 2024-06-03T04:53:24.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.34744 2024-06-03T04:53:24.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:53:24.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:53:24.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-06-03T04:53:24.847 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-06-03T04:53:24.847 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-06-03T04:53:24.860 INFO:tasks.workunit:Running workunit misc/ok-to-stop.sh... 2024-06-03T04:53:24.860 DEBUG:teuthology.orchestra.run.smithi088:workunit test misc/ok-to-stop.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dfcc4b3c2c55532c4c04fa47551c9df1dffdf746 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh 2024-06-03T04:53:24.918 INFO:tasks.workunit.client.0.smithi088.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-06-03T04:53:24.927 INFO:tasks.workunit.client.0.smithi088.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-06-03T04:53:24.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-03T04:53:24.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-03T04:53:24.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-06-03T04:53:24.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-06-03T04:53:24.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-06-03T04:53:24.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-06-03T04:53:24.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-06-03T04:53:24.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-06-03T04:53:24.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/ok-to-stop 2024-06-03T04:53:24.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:6: run: local dir=td/ok-to-stop 2024-06-03T04:53:24.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:7: run: shift 2024-06-03T04:53:24.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:9: run: export CEPH_MON_A=127.0.0.1:7150 2024-06-03T04:53:24.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:9: run: CEPH_MON_A=127.0.0.1:7150 2024-06-03T04:53:24.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:10: run: export CEPH_MON_B=127.0.0.1:7151 2024-06-03T04:53:24.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:10: run: CEPH_MON_B=127.0.0.1:7151 2024-06-03T04:53:24.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:11: run: export CEPH_MON_C=127.0.0.1:7152 2024-06-03T04:53:24.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:11: run: CEPH_MON_C=127.0.0.1:7152 2024-06-03T04:53:24.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:12: run: export CEPH_MON_D=127.0.0.1:7153 2024-06-03T04:53:24.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:12: run: CEPH_MON_D=127.0.0.1:7153 2024-06-03T04:53:24.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:13: run: export CEPH_MON_E=127.0.0.1:7154 2024-06-03T04:53:24.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:13: run: CEPH_MON_E=127.0.0.1:7154 2024-06-03T04:53:24.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:14: run: export CEPH_ARGS 2024-06-03T04:53:24.929 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:15: run: uuidgen 2024-06-03T04:53:24.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:15: run: CEPH_ARGS+='--fsid=cfc566bc-c3cf-46dc-a040-e327d99dd7fe --auth-supported=none ' 2024-06-03T04:53:24.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:16: run: export 'ORIG_CEPH_ARGS=--fsid=cfc566bc-c3cf-46dc-a040-e327d99dd7fe --auth-supported=none ' 2024-06-03T04:53:24.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:16: run: ORIG_CEPH_ARGS='--fsid=cfc566bc-c3cf-46dc-a040-e327d99dd7fe --auth-supported=none ' 2024-06-03T04:53:24.929 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: set 2024-06-03T04:53:24.929 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-06-03T04:53:24.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: local 'funcs=TEST_0_mds 2024-06-03T04:53:24.931 INFO:tasks.workunit.client.0.smithi088.stderr:TEST_0_osd 2024-06-03T04:53:24.931 INFO:tasks.workunit.client.0.smithi088.stderr:TEST_1_mon_checks 2024-06-03T04:53:24.931 INFO:tasks.workunit.client.0.smithi088.stderr:TEST_2_mons_checks 2024-06-03T04:53:24.931 INFO:tasks.workunit.client.0.smithi088.stderr:TEST_3_mons_checks 2024-06-03T04:53:24.931 INFO:tasks.workunit.client.0.smithi088.stderr:TEST_4_mons_checks 2024-06-03T04:53:24.931 INFO:tasks.workunit.client.0.smithi088.stderr:TEST_5_mons_checks' 2024-06-03T04:53:24.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-06-03T04:53:24.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-06-03T04:53:24.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-06-03T04:53:24.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-06-03T04:53:24.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-06-03T04:53:24.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T04:53:24.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-06-03T04:53:24.931 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:53:24.932 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:53:24.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:53:24.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:53:24.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:53:24.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:53:24.932 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:53:24.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:53:24.933 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:53:24.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:53:24.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:53:24.934 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:53:24.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:53:24.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:53:24.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:53:24.936 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:53:24.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:53:24.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T04:53:24.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-06-03T04:53:24.939 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:53:24.939 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:53:24.939 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:53:24.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47366 2024-06-03T04:53:24.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:53:24.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:53:24.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-06-03T04:53:24.941 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T04:53:24.941 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:53:24.941 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:53:24.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.47366 2024-06-03T04:53:24.944 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T04:53:24.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-06-03T04:53:24.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-06-03T04:53:24.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T04:53:24.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-06-03T04:53:24.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_0_mds td/ok-to-stop 2024-06-03T04:53:24.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:208: TEST_0_mds: local dir=td/ok-to-stop 2024-06-03T04:53:24.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:210: TEST_0_mds: CEPH_ARGS='--fsid=cfc566bc-c3cf-46dc-a040-e327d99dd7fe --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-06-03T04:53:24.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:212: TEST_0_mds: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-06-03T04:53:24.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-06-03T04:53:24.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T04:53:24.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T04:53:24.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T04:53:24.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-06-03T04:53:24.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-06-03T04:53:25.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T04:53:25.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:53:25.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:53:25.350 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:53:25.350 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:53:25.350 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:53:25.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47366/$cluster-$name.asok' 2024-06-03T04:53:25.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47366/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-06-03T04:53:25.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T04:53:25.579 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T04:53:25.580 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:53:25.580 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T04:53:25.580 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T04:53:25.580 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T04:53:25.580 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T04:53:25.581 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T04:53:25.581 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T04:53:25.582 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:53:25.582 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:53:25.582 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:53:25.582 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47366/ceph-mon.a.asok 2024-06-03T04:53:25.582 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:53:25.582 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47366/ceph-mon.a.asok config get fsid 2024-06-03T04:53:25.693 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T04:53:25.693 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:53:25.693 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T04:53:25.693 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T04:53:25.694 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T04:53:25.694 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T04:53:25.694 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T04:53:25.694 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T04:53:25.694 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:53:25.694 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:53:25.694 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:53:25.695 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47366/ceph-mon.a.asok 2024-06-03T04:53:25.695 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:53:25.696 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47366/ceph-mon.a.asok config get mon_host 2024-06-03T04:53:25.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:213: TEST_0_mds: run_mgr td/ok-to-stop x 2024-06-03T04:53:25.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ok-to-stop 2024-06-03T04:53:25.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T04:53:25.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T04:53:25.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T04:53:25.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ok-to-stop/x 2024-06-03T04:53:25.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T04:53:26.704 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T04:53:26.704 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:53:26.704 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:53:26.704 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:53:26.704 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:53:26.705 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:53:26.705 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47366/$cluster-$name.asok' 2024-06-03T04:53:26.705 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T04:53:26.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ok-to-stop/x '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47366/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T04:53:26.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:214: TEST_0_mds: run_osd td/ok-to-stop 0 2024-06-03T04:53:26.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-06-03T04:53:26.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T04:53:26.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T04:53:26.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T04:53:26.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/0 2024-06-03T04:53:26.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cfc566bc-c3cf-46dc-a040-e327d99dd7fe --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-06-03T04:53:26.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T04:53:26.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T04:53:26.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T04:53:26.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/0' 2024-06-03T04:53:26.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/0/journal' 2024-06-03T04:53:26.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T04:53:26.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T04:53:26.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-06-03T04:53:26.734 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T04:53:26.734 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:53:26.734 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:53:26.734 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:53:26.734 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:53:26.734 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:53:26.734 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47366/$cluster-$name.asok' 2024-06-03T04:53:26.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.47366/$cluster-$name.asok' 2024-06-03T04:53:26.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T04:53:26.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T04:53:26.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T04:53:26.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-06-03T04:53:26.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-06-03T04:53:26.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T04:53:26.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T04:53:26.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T04:53:26.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T04:53:26.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T04:53:26.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T04:53:26.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/0 2024-06-03T04:53:26.736 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T04:53:26.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=16098a7b-0588-443e-b74a-d1be359f88e6 2024-06-03T04:53:26.736 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 16098a7b-0588-443e-b74a-d1be359f88e6 2024-06-03T04:53:26.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 16098a7b-0588-443e-b74a-d1be359f88e6' 2024-06-03T04:53:26.737 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T04:53:26.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBGTF1mGnCcLBAAwn+T/5PYHeOalK2cIICUtA== 2024-06-03T04:53:26.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBGTF1mGnCcLBAAwn+T/5PYHeOalK2cIICUtA=="}' 2024-06-03T04:53:26.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 16098a7b-0588-443e-b74a-d1be359f88e6 -i td/ok-to-stop/0/new.json 2024-06-03T04:53:27.040 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T04:53:27.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/0/new.json 2024-06-03T04:53:27.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=cfc566bc-c3cf-46dc-a040-e327d99dd7fe --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.47366/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBGTF1mGnCcLBAAwn+T/5PYHeOalK2cIICUtA== --osd-uuid 16098a7b-0588-443e-b74a-d1be359f88e6 2024-06-03T04:53:27.091 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:53:27.087+0000 7fe7528fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:53:27.109 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:53:27.103+0000 7fe7528fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:53:27.110 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:53:27.103+0000 7fe7528fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:53:27.111 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:53:27.107+0000 7fe7528fe7c0 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-06-03T04:53:27.111 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:53:27.107+0000 7fe7528fe7c0 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-06-03T04:53:27.111 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:53:27.107+0000 7fe7528fe7c0 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-06-03T04:53:27.149 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:53:27.143+0000 7fe7528fe7c0 -1 bluestore(td/ok-to-stop/0) _read_fsid unparsable uuid 2024-06-03T04:53:33.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/0/keyring 2024-06-03T04:53:33.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T04:53:33.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T04:53:33.368 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T04:53:33.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T04:53:33.916 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T04:53:33.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T04:53:33.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=cfc566bc-c3cf-46dc-a040-e327d99dd7fe --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.47366/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T04:53:33.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T04:53:33.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T04:53:33.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T04:53:33.945 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:53:33.939+0000 7f885b8317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:53:33.948 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:53:33.943+0000 7f885b8317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:53:33.949 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:53:33.943+0000 7f885b8317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:53:34.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T04:53:34.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T04:53:34.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T04:53:34.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T04:53:34.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T04:53:34.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:53:34.364 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T04:53:34.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T04:53:34.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:53:34.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:53:34.516 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:53:34.511+0000 7f885b8317c0 -1 Falling back to public interface 2024-06-03T04:53:34.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:53:35.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:53:35.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:53:35.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T04:53:35.771 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T04:53:35.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:53:35.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:53:36.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:53:36.997 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:53:36.991+0000 7f885b8317c0 -1 osd.0 0 log_to_monitors true 2024-06-03T04:53:37.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:53:37.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:53:37.189 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T04:53:37.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T04:53:37.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:53:37.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:53:37.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:53:38.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:53:38.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:53:38.679 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T04:53:38.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T04:53:38.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:53:38.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:53:39.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:53:40.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:53:40.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:53:40.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T04:53:40.216 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T04:53:40.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:53:40.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:53:40.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:53:41.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:53:41.638 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T04:53:41.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:53:41.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T04:53:41.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:53:41.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:53:42.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:53:43.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:53:43.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:53:43.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T04:53:43.146 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T04:53:43.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:53:43.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:53:43.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:53:44.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:53:44.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:53:44.607 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T04:53:44.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T04:53:44.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:53:44.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:53:45.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:53:46.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:53:46.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:53:46.171 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T04:53:46.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T04:53:46.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:53:46.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:53:46.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:53:47.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:53:47.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:53:47.591 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T04:53:47.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T04:53:47.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:53:47.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:53:48.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:53:49.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:53:49.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:53:49.132 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T04:53:49.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T04:53:49.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:53:49.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:53:49.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:53:50.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:53:50.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:53:50.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T04:53:50.555 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T04:53:50.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:53:50.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:53:51.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:53:52.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:53:52.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:53:52.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T04:53:52.404 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T04:53:52.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:53:52.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:53:53.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:53:54.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:53:54.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:53:54.186 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T04:53:54.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T04:53:54.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:53:54.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:53:54.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:53:55.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:53:55.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:53:55.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T04:53:55.827 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T04:53:55.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:53:55.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:53:56.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:53:57.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:53:57.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:53:57.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T04:53:57.712 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T04:53:57.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:53:57.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:53:58.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:53:59.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:53:59.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:53:59.535 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T04:53:59.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T04:53:59.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:53:59.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:54:00.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:54:01.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:54:01.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:54:01.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T04:54:01.589 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T04:54:01.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:54:01.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:54:02.028 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3615859603,v1:127.0.0.1:6803/3615859603] [v2:127.0.0.1:6804/3615859603,v1:127.0.0.1:6805/3615859603] exists,up 16098a7b-0588-443e-b74a-d1be359f88e6 2024-06-03T04:54:02.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T04:54:02.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T04:54:02.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T04:54:02.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:215: TEST_0_mds: run_mds td/ok-to-stop a 2024-06-03T04:54:02.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ok-to-stop 2024-06-03T04:54:02.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-06-03T04:54:02.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=a 2024-06-03T04:54:02.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-06-03T04:54:02.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ok-to-stop/a 2024-06-03T04:54:02.031 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-06-03T04:54:02.031 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:54:02.031 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:54:02.031 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:54:02.031 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:54:02.031 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:54:02.031 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47366/$cluster-$name.asok' 2024-06-03T04:54:02.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id a --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47366/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' 2024-06-03T04:54:02.099 INFO:tasks.workunit.client.0.smithi088.stdout:starting mds.a at 2024-06-03T04:54:02.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:217: TEST_0_mds: ceph osd pool create meta 1 2024-06-03T04:54:02.746 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'meta' already exists 2024-06-03T04:54:02.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:218: TEST_0_mds: ceph osd pool create data 1 2024-06-03T04:54:03.738 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'data' already exists 2024-06-03T04:54:03.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:219: TEST_0_mds: ceph fs new myfs meta data 2024-06-03T04:54:04.916 INFO:tasks.workunit.client.0.smithi088.stderr:filesystem 'myfs' already exists 2024-06-03T04:54:04.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:220: TEST_0_mds: sleep 5 2024-06-03T04:54:09.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:222: TEST_0_mds: ceph mds ok-to-stop a 2024-06-03T04:54:10.302 INFO:tasks.workunit.client.0.smithi088.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4153 and/or standby gids 2024-06-03T04:54:10.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:223: TEST_0_mds: ceph mds ok-to-stop a dne 2024-06-03T04:54:10.715 INFO:tasks.workunit.client.0.smithi088.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4153 and/or standby gids 2024-06-03T04:54:10.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:224: TEST_0_mds: ceph mds ok-to-stop dne 2024-06-03T04:54:11.212 INFO:tasks.workunit.client.0.smithi088.stderr:should be safe to stop [dne] 2024-06-03T04:54:11.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:226: TEST_0_mds: run_mds td/ok-to-stop b 2024-06-03T04:54:11.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ok-to-stop 2024-06-03T04:54:11.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-06-03T04:54:11.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=b 2024-06-03T04:54:11.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-06-03T04:54:11.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ok-to-stop/b 2024-06-03T04:54:11.230 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-06-03T04:54:11.230 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:54:11.230 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:54:11.231 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:54:11.231 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:54:11.231 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:54:11.231 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47366/$cluster-$name.asok' 2024-06-03T04:54:11.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id b --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47366/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' 2024-06-03T04:54:11.254 INFO:tasks.workunit.client.0.smithi088.stdout:starting mds.b at 2024-06-03T04:54:11.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:227: TEST_0_mds: sleep 5 2024-06-03T04:54:16.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:229: TEST_0_mds: ceph mds ok-to-stop a 2024-06-03T04:54:16.787 INFO:tasks.workunit.client.0.smithi088.stderr:should be safe to stop [a] 2024-06-03T04:54:16.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:230: TEST_0_mds: ceph mds ok-to-stop b 2024-06-03T04:54:17.378 INFO:tasks.workunit.client.0.smithi088.stderr:should be safe to stop [b] 2024-06-03T04:54:17.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:231: TEST_0_mds: ceph mds ok-to-stop a b 2024-06-03T04:54:17.766 INFO:tasks.workunit.client.0.smithi088.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4153 and/or standby gids 4167 2024-06-03T04:54:17.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:232: TEST_0_mds: ceph mds ok-to-stop a dne1 dne2 2024-06-03T04:54:18.301 INFO:tasks.workunit.client.0.smithi088.stderr:should be safe to stop [a,dne1,dne2] 2024-06-03T04:54:18.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:233: TEST_0_mds: ceph mds ok-to-stop b dne 2024-06-03T04:54:18.858 INFO:tasks.workunit.client.0.smithi088.stderr:should be safe to stop [b,dne] 2024-06-03T04:54:18.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:234: TEST_0_mds: ceph mds ok-to-stop a b dne 2024-06-03T04:54:19.290 INFO:tasks.workunit.client.0.smithi088.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4153 and/or standby gids 4167 2024-06-03T04:54:19.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:235: TEST_0_mds: ceph mds ok-to-stop dne1 dne2 2024-06-03T04:54:19.837 INFO:tasks.workunit.client.0.smithi088.stderr:should be safe to stop [dne1,dne2] 2024-06-03T04:54:19.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:237: TEST_0_mds: kill_daemons td/ok-to-stop KILL mds.a 2024-06-03T04:54:19.853 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:54:19.853 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:54:19.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:54:19.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:54:19.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:54:19.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:54:19.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-06-03T04:54:19.960 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:54:19.960 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:54:19.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:54:19.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:54:19.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:54:20.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:54:20.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-06-03T04:54:20.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-06-03T04:54:20.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T04:54:20.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-06-03T04:54:20.072 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:54:20.072 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:54:20.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:54:20.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:54:20.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:54:20.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:54:20.077 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:54:20.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:54:20.078 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:54:20.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:54:20.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:54:20.079 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:54:20.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:54:20.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:54:20.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:54:20.080 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:54:20.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:54:20.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T04:54:20.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-06-03T04:54:20.088 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:54:20.088 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:54:20.088 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:54:20.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47366 2024-06-03T04:54:20.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:54:20.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:54:20.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-06-03T04:54:20.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-06-03T04:54:20.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-06-03T04:54:20.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-06-03T04:54:20.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-06-03T04:54:20.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T04:54:20.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-06-03T04:54:20.091 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:54:20.091 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:54:20.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:54:20.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:54:20.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:54:20.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:54:20.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:54:20.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:54:20.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:54:20.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:54:20.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:54:20.093 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:54:20.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:54:20.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:54:20.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:54:20.095 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:54:20.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:54:20.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T04:54:20.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-06-03T04:54:20.097 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:54:20.097 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:54:20.097 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:54:20.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47366 2024-06-03T04:54:20.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:54:20.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:54:20.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-06-03T04:54:20.099 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T04:54:20.099 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:54:20.099 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:54:20.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.47366 2024-06-03T04:54:20.101 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T04:54:20.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T04:54:20.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T04:54:20.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-06-03T04:54:20.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_0_osd td/ok-to-stop 2024-06-03T04:54:20.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:241: TEST_0_osd: local dir=td/ok-to-stop 2024-06-03T04:54:20.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:243: TEST_0_osd: CEPH_ARGS='--fsid=cfc566bc-c3cf-46dc-a040-e327d99dd7fe --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-06-03T04:54:20.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:245: TEST_0_osd: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-06-03T04:54:20.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-06-03T04:54:20.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T04:54:20.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T04:54:20.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T04:54:20.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-06-03T04:54:20.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-06-03T04:54:20.510 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T04:54:20.511 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:54:20.511 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:54:20.511 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:54:20.511 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:54:20.511 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:54:20.511 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47366/$cluster-$name.asok' 2024-06-03T04:54:20.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47366/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-06-03T04:54:20.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T04:54:20.707 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T04:54:20.707 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:54:20.707 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T04:54:20.707 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T04:54:20.708 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T04:54:20.708 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T04:54:20.708 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T04:54:20.708 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T04:54:20.709 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:54:20.709 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:54:20.709 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:54:20.709 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47366/ceph-mon.a.asok 2024-06-03T04:54:20.710 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:54:20.710 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47366/ceph-mon.a.asok config get fsid 2024-06-03T04:54:20.821 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T04:54:20.821 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:54:20.821 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T04:54:20.821 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T04:54:20.822 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T04:54:20.822 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T04:54:20.822 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T04:54:20.822 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T04:54:20.822 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:54:20.822 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:54:20.823 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:54:20.823 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47366/ceph-mon.a.asok 2024-06-03T04:54:20.823 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:54:20.823 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47366/ceph-mon.a.asok config get mon_host 2024-06-03T04:54:20.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:246: TEST_0_osd: run_mgr td/ok-to-stop x 2024-06-03T04:54:20.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ok-to-stop 2024-06-03T04:54:20.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T04:54:20.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T04:54:20.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T04:54:20.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ok-to-stop/x 2024-06-03T04:54:20.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T04:54:21.835 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T04:54:21.835 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:54:21.836 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:54:21.836 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:54:21.836 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:54:21.836 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:54:21.836 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47366/$cluster-$name.asok' 2024-06-03T04:54:21.837 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T04:54:21.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ok-to-stop/x '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47366/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T04:54:21.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:247: TEST_0_osd: run_osd td/ok-to-stop 0 2024-06-03T04:54:21.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-06-03T04:54:21.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T04:54:21.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T04:54:21.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T04:54:21.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/0 2024-06-03T04:54:21.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cfc566bc-c3cf-46dc-a040-e327d99dd7fe --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-06-03T04:54:21.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T04:54:21.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T04:54:21.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T04:54:21.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/0' 2024-06-03T04:54:21.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/0/journal' 2024-06-03T04:54:21.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T04:54:21.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T04:54:21.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-06-03T04:54:21.871 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T04:54:21.871 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:54:21.871 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:54:21.871 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:54:21.871 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:54:21.871 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:54:21.871 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47366/$cluster-$name.asok' 2024-06-03T04:54:21.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.47366/$cluster-$name.asok' 2024-06-03T04:54:21.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T04:54:21.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T04:54:21.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T04:54:21.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-06-03T04:54:21.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-06-03T04:54:21.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T04:54:21.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T04:54:21.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T04:54:21.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T04:54:21.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T04:54:21.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T04:54:21.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/0 2024-06-03T04:54:21.873 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T04:54:21.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3a880e4f-d6cf-4a07-8900-e7cce17a354b 2024-06-03T04:54:21.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3a880e4f-d6cf-4a07-8900-e7cce17a354b' 2024-06-03T04:54:21.873 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 3a880e4f-d6cf-4a07-8900-e7cce17a354b 2024-06-03T04:54:21.874 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T04:54:21.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB9TF1mix7GNBAA2MB223SVDDScawt5UBfa5A== 2024-06-03T04:54:21.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB9TF1mix7GNBAA2MB223SVDDScawt5UBfa5A=="}' 2024-06-03T04:54:21.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3a880e4f-d6cf-4a07-8900-e7cce17a354b -i td/ok-to-stop/0/new.json 2024-06-03T04:54:22.202 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T04:54:22.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/0/new.json 2024-06-03T04:54:22.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=cfc566bc-c3cf-46dc-a040-e327d99dd7fe --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.47366/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB9TF1mix7GNBAA2MB223SVDDScawt5UBfa5A== --osd-uuid 3a880e4f-d6cf-4a07-8900-e7cce17a354b 2024-06-03T04:54:22.249 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:54:22.243+0000 7f5ee3a7f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:54:22.287 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:54:22.283+0000 7f5ee3a7f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:54:22.288 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:54:22.283+0000 7f5ee3a7f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:54:22.289 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:54:22.283+0000 7f5ee3a7f7c0 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-06-03T04:54:22.289 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:54:22.283+0000 7f5ee3a7f7c0 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-06-03T04:54:22.289 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:54:22.283+0000 7f5ee3a7f7c0 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-06-03T04:54:22.327 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:54:22.323+0000 7f5ee3a7f7c0 -1 bluestore(td/ok-to-stop/0) _read_fsid unparsable uuid 2024-06-03T04:54:28.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/0/keyring 2024-06-03T04:54:28.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T04:54:28.659 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T04:54:28.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T04:54:28.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T04:54:29.212 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T04:54:29.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T04:54:29.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=cfc566bc-c3cf-46dc-a040-e327d99dd7fe --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.47366/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T04:54:29.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T04:54:29.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T04:54:29.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T04:54:29.242 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:54:29.235+0000 7f02059147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:54:29.286 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:54:29.279+0000 7f02059147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:54:29.287 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:54:29.283+0000 7f02059147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:54:29.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T04:54:29.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T04:54:29.661 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T04:54:29.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T04:54:29.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T04:54:29.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T04:54:29.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:54:29.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T04:54:29.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:54:29.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:54:29.869 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:54:29.863+0000 7f02059147c0 -1 Falling back to public interface 2024-06-03T04:54:30.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:54:31.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:54:31.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:54:31.083 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T04:54:31.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T04:54:31.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:54:31.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:54:31.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:54:32.476 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:54:32.471+0000 7f02059147c0 -1 osd.0 0 log_to_monitors true 2024-06-03T04:54:32.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:54:32.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:54:32.502 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T04:54:32.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T04:54:32.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:54:32.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:54:33.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:54:34.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:54:34.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:54:34.188 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T04:54:34.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T04:54:34.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:54:34.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:54:34.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:54:35.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:54:35.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:54:35.667 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T04:54:35.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T04:54:35.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:54:35.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:54:36.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:54:37.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:54:37.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:54:37.105 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T04:54:37.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T04:54:37.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:54:37.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:54:37.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:54:38.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:54:38.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:54:38.644 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T04:54:38.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T04:54:38.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:54:38.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:54:39.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:54:40.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:54:40.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:54:40.196 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T04:54:40.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T04:54:40.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:54:40.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:54:40.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:54:41.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:54:41.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:54:41.625 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T04:54:41.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T04:54:41.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:54:41.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:54:42.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:54:43.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:54:43.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:54:43.088 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T04:54:43.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T04:54:43.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:54:43.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:54:43.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:54:44.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:54:44.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:54:44.617 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T04:54:44.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T04:54:44.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:54:44.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:54:45.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:54:46.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:54:46.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:54:46.028 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T04:54:46.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T04:54:46.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:54:46.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:54:46.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:54:47.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:54:47.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:54:47.891 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T04:54:47.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T04:54:47.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:54:47.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:54:48.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:54:49.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:54:49.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:54:49.718 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T04:54:49.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T04:54:49.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:54:49.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:54:50.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:54:51.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:54:51.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:54:51.580 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T04:54:51.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T04:54:51.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:54:51.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:54:52.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:54:53.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:54:53.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:54:53.326 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T04:54:53.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T04:54:53.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:54:53.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:54:53.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:54:54.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:54:54.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:54:54.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T04:54:54.868 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T04:54:54.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:54:54.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:54:55.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:54:56.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:54:56.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:54:56.664 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T04:54:56.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T04:54:56.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:54:56.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T04:54:57.109 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1408592183,v1:127.0.0.1:6803/1408592183] [v2:127.0.0.1:6804/1408592183,v1:127.0.0.1:6805/1408592183] exists,up 3a880e4f-d6cf-4a07-8900-e7cce17a354b 2024-06-03T04:54:57.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T04:54:57.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T04:54:57.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T04:54:57.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:248: TEST_0_osd: run_osd td/ok-to-stop 1 2024-06-03T04:54:57.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-06-03T04:54:57.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T04:54:57.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-03T04:54:57.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T04:54:57.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/1 2024-06-03T04:54:57.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cfc566bc-c3cf-46dc-a040-e327d99dd7fe --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-06-03T04:54:57.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T04:54:57.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T04:54:57.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T04:54:57.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/1' 2024-06-03T04:54:57.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/1/journal' 2024-06-03T04:54:57.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T04:54:57.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T04:54:57.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-06-03T04:54:57.123 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T04:54:57.123 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:54:57.123 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:54:57.123 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:54:57.123 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:54:57.123 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:54:57.124 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47366/$cluster-$name.asok' 2024-06-03T04:54:57.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.47366/$cluster-$name.asok' 2024-06-03T04:54:57.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T04:54:57.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T04:54:57.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T04:54:57.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-06-03T04:54:57.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-06-03T04:54:57.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T04:54:57.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T04:54:57.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T04:54:57.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T04:54:57.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T04:54:57.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T04:54:57.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/1 2024-06-03T04:54:57.125 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T04:54:57.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7680ad8c-91ea-419f-934e-f82f614f7aa4 2024-06-03T04:54:57.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 7680ad8c-91ea-419f-934e-f82f614f7aa4' 2024-06-03T04:54:57.126 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 7680ad8c-91ea-419f-934e-f82f614f7aa4 2024-06-03T04:54:57.127 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T04:54:57.127 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQChTF1mIQ9lBxAAK+L3mc9k6qykS/ky1+R4CQ== 2024-06-03T04:54:57.127 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQChTF1mIQ9lBxAAK+L3mc9k6qykS/ky1+R4CQ=="}' 2024-06-03T04:54:57.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7680ad8c-91ea-419f-934e-f82f614f7aa4 -i td/ok-to-stop/1/new.json 2024-06-03T04:54:57.785 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T04:54:57.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/1/new.json 2024-06-03T04:54:57.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=cfc566bc-c3cf-46dc-a040-e327d99dd7fe --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/1 --osd-journal=td/ok-to-stop/1/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.47366/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQChTF1mIQ9lBxAAK+L3mc9k6qykS/ky1+R4CQ== --osd-uuid 7680ad8c-91ea-419f-934e-f82f614f7aa4 2024-06-03T04:54:57.834 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:54:57.827+0000 7f82d799f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:54:57.875 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:54:57.871+0000 7f82d799f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:54:57.876 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:54:57.871+0000 7f82d799f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:54:57.877 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:54:57.871+0000 7f82d799f7c0 -1 bluestore(td/ok-to-stop/1/block) _read_bdev_label failed to open td/ok-to-stop/1/block: (2) No such file or directory 2024-06-03T04:54:57.877 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:54:57.871+0000 7f82d799f7c0 -1 bluestore(td/ok-to-stop/1/block) _read_bdev_label failed to open td/ok-to-stop/1/block: (2) No such file or directory 2024-06-03T04:54:57.877 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:54:57.871+0000 7f82d799f7c0 -1 bluestore(td/ok-to-stop/1/block) _read_bdev_label failed to open td/ok-to-stop/1/block: (2) No such file or directory 2024-06-03T04:54:57.914 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:54:57.907+0000 7f82d799f7c0 -1 bluestore(td/ok-to-stop/1) _read_fsid unparsable uuid 2024-06-03T04:55:04.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/1/keyring 2024-06-03T04:55:04.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T04:55:04.537 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-06-03T04:55:04.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-03T04:55:04.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T04:55:05.083 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-06-03T04:55:05.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-03T04:55:05.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=cfc566bc-c3cf-46dc-a040-e327d99dd7fe --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/1 --osd-journal=td/ok-to-stop/1/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.47366/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T04:55:05.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T04:55:05.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T04:55:05.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T04:55:05.107 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:55:05.103+0000 7fec5faee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:55:05.121 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:55:05.115+0000 7fec5faee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:55:05.122 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:55:05.119+0000 7fec5faee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:55:05.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-03T04:55:05.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T04:55:05.487 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T04:55:05.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T04:55:05.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T04:55:05.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T04:55:05.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:55:05.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T04:55:05.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:55:05.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:55:05.704 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:55:05.699+0000 7fec5faee7c0 -1 Falling back to public interface 2024-06-03T04:55:05.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:55:06.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:55:06.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:55:06.918 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T04:55:06.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T04:55:06.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:55:06.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:55:07.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:55:08.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:55:08.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:55:08.334 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T04:55:08.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T04:55:08.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:55:08.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:55:08.359 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:55:08.355+0000 7fec5faee7c0 -1 osd.1 0 log_to_monitors true 2024-06-03T04:55:08.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:55:09.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:55:09.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:55:09.741 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T04:55:09.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T04:55:09.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:55:09.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:55:10.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:55:11.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:55:11.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:55:11.149 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T04:55:11.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T04:55:11.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:55:11.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:55:11.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:55:12.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:55:12.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:55:12.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T04:55:12.727 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T04:55:12.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:55:12.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:55:13.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:55:14.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:55:14.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:55:14.255 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T04:55:14.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T04:55:14.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:55:14.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:55:14.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:55:15.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:55:15.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:55:15.748 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T04:55:15.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T04:55:15.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:55:15.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:55:16.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:55:17.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:55:17.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:55:17.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T04:55:17.168 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T04:55:17.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:55:17.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:55:17.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:55:18.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:55:18.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:55:18.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T04:55:18.724 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T04:55:18.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:55:18.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:55:19.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:55:20.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:55:20.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:55:20.227 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T04:55:20.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T04:55:20.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:55:20.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:55:20.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:55:21.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:55:21.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:55:21.743 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T04:55:21.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T04:55:21.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:55:21.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:55:22.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:55:23.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:55:23.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:55:23.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T04:55:23.207 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T04:55:23.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:55:23.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:55:23.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:55:24.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:55:24.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:55:24.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T04:55:24.857 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T04:55:24.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:55:24.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:55:25.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:55:26.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:55:26.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:55:26.927 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T04:55:26.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T04:55:26.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:55:26.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:55:27.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:55:28.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:55:28.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:55:28.844 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T04:55:28.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T04:55:28.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:55:28.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:55:29.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:55:30.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:55:30.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:55:30.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T04:55:30.707 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T04:55:30.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:55:30.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:55:31.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:55:32.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:55:32.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:55:32.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T04:55:32.204 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T04:55:32.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:55:32.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:55:33.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:55:34.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:55:34.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:55:34.363 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-06-03T04:55:34.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T04:55:34.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:55:34.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T04:55:34.798 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3185486361,v1:127.0.0.1:6811/3185486361] [v2:127.0.0.1:6812/3185486361,v1:127.0.0.1:6813/3185486361] exists,up 7680ad8c-91ea-419f-934e-f82f614f7aa4 2024-06-03T04:55:34.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T04:55:34.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T04:55:34.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T04:55:34.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:249: TEST_0_osd: run_osd td/ok-to-stop 2 2024-06-03T04:55:34.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-06-03T04:55:34.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T04:55:34.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-03T04:55:34.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T04:55:34.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/2 2024-06-03T04:55:34.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cfc566bc-c3cf-46dc-a040-e327d99dd7fe --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-06-03T04:55:34.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T04:55:34.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T04:55:34.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T04:55:34.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/2' 2024-06-03T04:55:34.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/2/journal' 2024-06-03T04:55:34.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T04:55:34.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T04:55:34.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-06-03T04:55:34.803 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T04:55:34.803 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:55:34.803 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:55:34.803 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:55:34.803 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:55:34.803 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:55:34.803 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47366/$cluster-$name.asok' 2024-06-03T04:55:34.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.47366/$cluster-$name.asok' 2024-06-03T04:55:34.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T04:55:34.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T04:55:34.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T04:55:34.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-06-03T04:55:34.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-06-03T04:55:34.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T04:55:34.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T04:55:34.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T04:55:34.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T04:55:34.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T04:55:34.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T04:55:34.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/2 2024-06-03T04:55:34.804 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T04:55:34.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bade1d29-9fa9-4afe-8e39-b1a1d5b726e4 2024-06-03T04:55:34.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 bade1d29-9fa9-4afe-8e39-b1a1d5b726e4' 2024-06-03T04:55:34.805 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 bade1d29-9fa9-4afe-8e39-b1a1d5b726e4 2024-06-03T04:55:34.805 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T04:55:34.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDGTF1m4GffMBAA9JZDpO9MsznkRbxguFg8hw== 2024-06-03T04:55:34.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDGTF1m4GffMBAA9JZDpO9MsznkRbxguFg8hw=="}' 2024-06-03T04:55:34.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bade1d29-9fa9-4afe-8e39-b1a1d5b726e4 -i td/ok-to-stop/2/new.json 2024-06-03T04:55:35.300 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T04:55:35.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/2/new.json 2024-06-03T04:55:35.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=cfc566bc-c3cf-46dc-a040-e327d99dd7fe --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/2 --osd-journal=td/ok-to-stop/2/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.47366/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDGTF1m4GffMBAA9JZDpO9MsznkRbxguFg8hw== --osd-uuid bade1d29-9fa9-4afe-8e39-b1a1d5b726e4 2024-06-03T04:55:35.341 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:55:35.335+0000 7f0254df47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:55:35.381 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:55:35.375+0000 7f0254df47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:55:35.383 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:55:35.379+0000 7f0254df47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:55:35.383 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:55:35.379+0000 7f0254df47c0 -1 bluestore(td/ok-to-stop/2/block) _read_bdev_label failed to open td/ok-to-stop/2/block: (2) No such file or directory 2024-06-03T04:55:35.383 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:55:35.379+0000 7f0254df47c0 -1 bluestore(td/ok-to-stop/2/block) _read_bdev_label failed to open td/ok-to-stop/2/block: (2) No such file or directory 2024-06-03T04:55:35.383 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:55:35.379+0000 7f0254df47c0 -1 bluestore(td/ok-to-stop/2/block) _read_bdev_label failed to open td/ok-to-stop/2/block: (2) No such file or directory 2024-06-03T04:55:35.428 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:55:35.423+0000 7f0254df47c0 -1 bluestore(td/ok-to-stop/2) _read_fsid unparsable uuid 2024-06-03T04:55:42.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/2/keyring 2024-06-03T04:55:42.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T04:55:42.195 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-06-03T04:55:42.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-03T04:55:42.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T04:55:42.732 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-06-03T04:55:42.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-03T04:55:42.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=cfc566bc-c3cf-46dc-a040-e327d99dd7fe --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/2 --osd-journal=td/ok-to-stop/2/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.47366/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T04:55:42.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T04:55:42.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T04:55:42.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T04:55:42.761 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:55:42.755+0000 7f1dda4887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:55:42.763 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:55:42.759+0000 7f1dda4887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:55:42.764 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:55:42.759+0000 7f1dda4887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:55:43.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-03T04:55:43.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T04:55:43.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T04:55:43.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T04:55:43.192 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T04:55:43.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T04:55:43.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:55:43.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T04:55:43.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:55:43.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:55:43.324 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:55:43.319+0000 7f1dda4887c0 -1 Falling back to public interface 2024-06-03T04:55:43.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:55:44.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:55:44.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:55:44.626 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T04:55:44.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T04:55:44.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:55:44.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:55:45.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:55:45.987 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:55:45.983+0000 7f1dda4887c0 -1 osd.2 0 log_to_monitors true 2024-06-03T04:55:46.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:55:46.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:55:46.055 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T04:55:46.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T04:55:46.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:55:46.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:55:46.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:55:47.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:55:47.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:55:47.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T04:55:47.845 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T04:55:47.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:55:47.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:55:48.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:55:49.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:55:49.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:55:49.307 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T04:55:49.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T04:55:49.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:55:49.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:55:49.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:55:50.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:55:50.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:55:50.890 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T04:55:50.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T04:55:50.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:55:50.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:55:51.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:55:52.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:55:52.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:55:52.404 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T04:55:52.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T04:55:52.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:55:52.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:55:53.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:55:54.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:55:54.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:55:54.007 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T04:55:54.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T04:55:54.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:55:54.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:55:54.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:55:55.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:55:55.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:55:55.459 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T04:55:55.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T04:55:55.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:55:55.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:55:56.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:55:57.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:55:57.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:55:57.025 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T04:55:57.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T04:55:57.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:55:57.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:55:57.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:55:58.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:55:58.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:55:58.498 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T04:55:58.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T04:55:58.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:55:58.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:55:58.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:55:59.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:55:59.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:55:59.990 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T04:55:59.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T04:55:59.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:55:59.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:56:00.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:56:01.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:56:01.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:56:01.772 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T04:56:01.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T04:56:01.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:56:01.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:56:02.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:56:03.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:56:03.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:56:03.591 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T04:56:03.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T04:56:03.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:56:03.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:56:04.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:56:05.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:56:05.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:56:05.383 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T04:56:05.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T04:56:05.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:56:05.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:56:06.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:56:07.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:56:07.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:56:07.652 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T04:56:07.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T04:56:07.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:56:07.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:56:08.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:56:09.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:56:09.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:56:09.380 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T04:56:09.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T04:56:09.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:56:09.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:56:10.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:56:11.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:56:11.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:56:11.282 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T04:56:11.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T04:56:11.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:56:11.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T04:56:11.850 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2271829324,v1:127.0.0.1:6819/2271829324] [v2:127.0.0.1:6820/2271829324,v1:127.0.0.1:6821/2271829324] exists,up bade1d29-9fa9-4afe-8e39-b1a1d5b726e4 2024-06-03T04:56:11.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T04:56:11.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T04:56:11.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T04:56:11.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:250: TEST_0_osd: run_osd td/ok-to-stop 3 2024-06-03T04:56:11.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-06-03T04:56:11.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T04:56:11.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-06-03T04:56:11.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T04:56:11.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/3 2024-06-03T04:56:11.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cfc566bc-c3cf-46dc-a040-e327d99dd7fe --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-06-03T04:56:11.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T04:56:11.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T04:56:11.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T04:56:11.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/3' 2024-06-03T04:56:11.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/3/journal' 2024-06-03T04:56:11.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T04:56:11.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T04:56:11.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-06-03T04:56:11.855 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T04:56:11.855 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:56:11.855 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:56:11.855 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:56:11.855 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:56:11.855 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:56:11.855 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47366/$cluster-$name.asok' 2024-06-03T04:56:11.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.47366/$cluster-$name.asok' 2024-06-03T04:56:11.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T04:56:11.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T04:56:11.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T04:56:11.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-06-03T04:56:11.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-06-03T04:56:11.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T04:56:11.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T04:56:11.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T04:56:11.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T04:56:11.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T04:56:11.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T04:56:11.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/3 2024-06-03T04:56:11.857 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T04:56:11.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=de8d882b-0c7c-4752-9b85-faf0256215d6 2024-06-03T04:56:11.857 INFO:tasks.workunit.client.0.smithi088.stdout:add osd3 de8d882b-0c7c-4752-9b85-faf0256215d6 2024-06-03T04:56:11.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 de8d882b-0c7c-4752-9b85-faf0256215d6' 2024-06-03T04:56:11.857 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T04:56:11.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDrTF1mox/OMxAAsiScDw0YR7Knkj/QXW98FQ== 2024-06-03T04:56:11.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDrTF1mox/OMxAAsiScDw0YR7Knkj/QXW98FQ=="}' 2024-06-03T04:56:11.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new de8d882b-0c7c-4752-9b85-faf0256215d6 -i td/ok-to-stop/3/new.json 2024-06-03T04:56:12.526 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T04:56:12.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/3/new.json 2024-06-03T04:56:12.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=cfc566bc-c3cf-46dc-a040-e327d99dd7fe --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/3 --osd-journal=td/ok-to-stop/3/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.47366/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDrTF1mox/OMxAAsiScDw0YR7Knkj/QXW98FQ== --osd-uuid de8d882b-0c7c-4752-9b85-faf0256215d6 2024-06-03T04:56:12.582 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:56:12.579+0000 7f418a92c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:56:12.590 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:56:12.587+0000 7f418a92c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:56:12.592 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:56:12.587+0000 7f418a92c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:56:12.592 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:56:12.587+0000 7f418a92c7c0 -1 bluestore(td/ok-to-stop/3/block) _read_bdev_label failed to open td/ok-to-stop/3/block: (2) No such file or directory 2024-06-03T04:56:12.592 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:56:12.587+0000 7f418a92c7c0 -1 bluestore(td/ok-to-stop/3/block) _read_bdev_label failed to open td/ok-to-stop/3/block: (2) No such file or directory 2024-06-03T04:56:12.592 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:56:12.587+0000 7f418a92c7c0 -1 bluestore(td/ok-to-stop/3/block) _read_bdev_label failed to open td/ok-to-stop/3/block: (2) No such file or directory 2024-06-03T04:56:12.679 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:56:12.675+0000 7f418a92c7c0 -1 bluestore(td/ok-to-stop/3) _read_fsid unparsable uuid 2024-06-03T04:56:19.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/3/keyring 2024-06-03T04:56:19.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T04:56:19.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-06-03T04:56:19.271 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd3 key to auth repository 2024-06-03T04:56:19.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T04:56:19.817 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.3 2024-06-03T04:56:19.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-06-03T04:56:19.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=cfc566bc-c3cf-46dc-a040-e327d99dd7fe --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/3 --osd-journal=td/ok-to-stop/3/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.47366/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T04:56:19.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T04:56:19.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T04:56:19.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T04:56:19.846 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:56:19.839+0000 7fee1932a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:56:19.848 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:56:19.843+0000 7fee1932a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:56:19.849 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:56:19.843+0000 7fee1932a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T04:56:20.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-06-03T04:56:20.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T04:56:20.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-03T04:56:20.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T04:56:20.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T04:56:20.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:56:20.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T04:56:20.341 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T04:56:20.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:56:20.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T04:56:20.409 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:56:20.403+0000 7fee1932a7c0 -1 Falling back to public interface 2024-06-03T04:56:20.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:56:21.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:56:21.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:56:21.789 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T04:56:21.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T04:56:21.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:56:21.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T04:56:22.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:56:23.165 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T04:56:23.159+0000 7fee1932a7c0 -1 osd.3 0 log_to_monitors true 2024-06-03T04:56:23.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:56:23.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:56:23.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T04:56:23.191 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T04:56:23.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:56:23.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T04:56:23.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:56:24.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:56:24.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:56:24.964 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T04:56:24.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T04:56:24.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:56:24.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T04:56:25.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:56:26.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:56:26.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:56:26.533 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T04:56:26.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T04:56:26.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:56:26.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T04:56:27.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:56:28.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:56:28.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:56:28.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T04:56:28.104 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T04:56:28.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:56:28.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T04:56:28.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:56:29.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:56:29.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:56:29.521 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T04:56:29.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T04:56:29.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:56:29.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T04:56:29.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:56:30.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:56:30.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:56:30.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T04:56:30.923 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T04:56:30.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:56:30.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T04:56:31.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:56:32.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:56:32.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:56:32.470 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T04:56:32.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T04:56:32.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:56:32.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T04:56:32.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:56:33.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:56:33.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:56:33.891 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T04:56:33.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T04:56:33.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:56:33.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T04:56:34.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:56:35.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:56:35.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:56:35.360 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T04:56:35.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T04:56:35.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:56:35.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T04:56:35.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:56:36.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:56:36.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:56:36.820 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T04:56:36.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T04:56:36.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:56:36.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T04:56:37.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:56:38.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:56:38.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:56:38.294 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T04:56:38.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T04:56:38.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:56:38.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T04:56:39.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:56:40.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:56:40.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:56:40.082 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T04:56:40.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T04:56:40.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:56:40.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T04:56:40.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:56:41.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:56:41.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:56:41.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T04:56:41.912 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T04:56:41.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:56:41.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T04:56:42.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:56:43.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:56:43.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:56:43.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T04:56:43.777 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T04:56:43.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:56:43.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T04:56:44.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:56:45.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:56:45.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:56:45.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T04:56:45.564 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T04:56:45.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:56:45.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T04:56:46.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:56:47.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:56:47.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:56:47.209 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T04:56:47.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T04:56:47.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:56:47.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T04:56:47.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T04:56:48.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T04:56:48.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T04:56:48.929 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-06-03T04:56:48.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T04:56:48.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T04:56:48.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-03T04:56:49.337 INFO:tasks.workunit.client.0.smithi088.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/4025760840,v1:127.0.0.1:6827/4025760840] [v2:127.0.0.1:6828/4025760840,v1:127.0.0.1:6829/4025760840] exists,up de8d882b-0c7c-4752-9b85-faf0256215d6 2024-06-03T04:56:49.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T04:56:49.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T04:56:49.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T04:56:49.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:252: TEST_0_osd: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2024-06-03T04:56:49.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:253: TEST_0_osd: ceph osd pool create ec erasure ec-profile 2024-06-03T04:56:50.795 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'ec' already exists 2024-06-03T04:56:51.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:255: TEST_0_osd: wait_for_clean 2024-06-03T04:56:51.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T04:56:51.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T04:56:51.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T04:56:51.065 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T04:56:51.065 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T04:56:51.065 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T04:56:51.065 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T04:56:51.065 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T04:56:51.065 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T04:56:51.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T04:56:51.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T04:56:51.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T04:56:51.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T04:56:51.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T04:56:51.066 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T04:56:51.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T04:56:51.655 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-06-03T04:56:51.655 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-06-03T04:56:51.655 INFO:tasks.workunit.client.0.smithi088.stderr:3' 2024-06-03T04:56:51.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T04:56:51.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T04:56:51.656 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T04:56:51.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836508 2024-06-03T04:56:51.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836508 2024-06-03T04:56:51.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508' 2024-06-03T04:56:51.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T04:56:51.926 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T04:56:52.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672981 2024-06-03T04:56:52.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672981 2024-06-03T04:56:52.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672981' 2024-06-03T04:56:52.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T04:56:52.151 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T04:56:52.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509459 2024-06-03T04:56:52.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509459 2024-06-03T04:56:52.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672981 2-64424509459' 2024-06-03T04:56:52.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T04:56:52.304 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T04:56:52.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345928 2024-06-03T04:56:52.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345928 2024-06-03T04:56:52.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672981 2-64424509459 3-85899345928' 2024-06-03T04:56:52.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T04:56:52.449 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836508 2024-06-03T04:56:52.449 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T04:56:52.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T04:56:52.450 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836508 2024-06-03T04:56:52.450 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T04:56:52.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836508 2024-06-03T04:56:52.452 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836508 2024-06-03T04:56:52.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-06-03T04:56:52.452 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T04:56:52.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836508 2024-06-03T04:56:52.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T04:56:52.951 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672981 2024-06-03T04:56:52.951 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T04:56:52.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T04:56:52.954 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T04:56:52.954 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672981 2024-06-03T04:56:52.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672981 2024-06-03T04:56:52.955 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672981 2024-06-03T04:56:52.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-06-03T04:56:52.955 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T04:56:53.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672981 -lt 42949672981 2024-06-03T04:56:53.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T04:56:53.369 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509459 2024-06-03T04:56:53.369 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T04:56:53.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T04:56:53.372 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T04:56:53.372 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509459 2024-06-03T04:56:53.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509459 2024-06-03T04:56:53.373 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509459 2024-06-03T04:56:53.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-06-03T04:56:53.374 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T04:56:53.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509459 -lt 64424509459 2024-06-03T04:56:53.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T04:56:53.815 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345928 2024-06-03T04:56:53.815 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T04:56:53.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T04:56:53.816 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345928 2024-06-03T04:56:53.817 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T04:56:53.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345928 2024-06-03T04:56:53.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345928' 2024-06-03T04:56:53.818 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 85899345928 2024-06-03T04:56:53.818 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T04:56:54.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345928 -lt 85899345928 2024-06-03T04:56:54.274 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T04:56:54.274 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T04:56:54.274 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T04:56:54.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-06-03T04:56:54.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T04:56:54.828 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T04:56:54.828 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T04:56:54.828 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T04:56:54.828 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T04:56:54.828 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T04:56:54.829 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T04:56:55.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-06-03T04:56:55.249 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T04:56:55.249 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T04:56:55.249 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T04:56:55.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 32 2024-06-03T04:56:55.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' -1 2024-06-03T04:56:55.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-03T04:56:55.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=8 2024-06-03T04:56:55.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T04:56:55.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-03T04:56:55.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T04:56:55.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T04:56:55.871 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T04:56:55.871 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T04:56:55.871 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T04:56:55.871 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T04:56:55.871 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T04:56:55.871 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T04:56:56.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-06-03T04:56:56.304 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T04:56:56.304 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T04:56:56.305 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T04:56:56.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 32 2024-06-03T04:56:56.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-06-03T04:56:56.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-03T04:56:56.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T04:56:56.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T04:56:56.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T04:56:56.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T04:56:56.831 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T04:56:56.831 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T04:56:57.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-03T04:56:57.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-03T04:56:57.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-03T04:56:57.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T04:56:57.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-03T04:56:57.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T04:56:57.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T04:56:57.586 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T04:56:57.586 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T04:56:57.586 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T04:56:57.586 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T04:56:57.586 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T04:56:57.587 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T04:56:57.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=28 2024-06-03T04:56:57.980 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T04:56:57.981 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T04:56:57.981 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T04:56:58.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 28 = 32 2024-06-03T04:56:58.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 28 '!=' 8 2024-06-03T04:56:58.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-03T04:56:58.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=28 2024-06-03T04:56:58.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T04:56:58.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-03T04:56:58.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T04:56:58.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T04:56:58.602 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T04:56:58.603 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T04:56:58.603 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T04:56:58.603 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T04:56:58.603 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T04:56:58.603 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T04:56:59.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-06-03T04:56:59.027 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T04:56:59.027 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T04:56:59.027 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T04:56:59.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-06-03T04:56:59.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T04:56:59.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T04:56:59.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:258: TEST_0_osd: ceph osd pool set ec min_size 3 2024-06-03T04:57:00.323 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 1 min_size to 3 2024-06-03T04:57:00.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:259: TEST_0_osd: wait_for_clean 2024-06-03T04:57:00.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T04:57:00.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T04:57:00.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T04:57:00.347 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T04:57:00.347 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T04:57:00.347 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T04:57:00.347 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T04:57:00.347 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T04:57:00.347 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T04:57:00.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T04:57:00.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T04:57:00.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T04:57:00.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T04:57:00.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T04:57:00.461 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T04:57:00.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T04:57:00.889 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-06-03T04:57:00.890 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-06-03T04:57:00.890 INFO:tasks.workunit.client.0.smithi088.stderr:3' 2024-06-03T04:57:00.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T04:57:00.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T04:57:00.890 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T04:57:01.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836512 2024-06-03T04:57:01.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836512 2024-06-03T04:57:01.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512' 2024-06-03T04:57:01.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T04:57:01.087 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T04:57:01.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672985 2024-06-03T04:57:01.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672985 2024-06-03T04:57:01.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512 1-42949672985' 2024-06-03T04:57:01.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T04:57:01.239 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T04:57:01.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509462 2024-06-03T04:57:01.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509462 2024-06-03T04:57:01.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512 1-42949672985 2-64424509462' 2024-06-03T04:57:01.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T04:57:01.387 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T04:57:01.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345932 2024-06-03T04:57:01.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345932 2024-06-03T04:57:01.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512 1-42949672985 2-64424509462 3-85899345932' 2024-06-03T04:57:01.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T04:57:01.537 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836512 2024-06-03T04:57:01.537 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T04:57:01.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T04:57:01.538 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836512 2024-06-03T04:57:01.538 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T04:57:01.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836512 2024-06-03T04:57:01.539 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836512 2024-06-03T04:57:01.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-06-03T04:57:01.540 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T04:57:01.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836510 -lt 21474836512 2024-06-03T04:57:01.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T04:57:02.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T04:57:02.941 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T04:57:03.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836512 -lt 21474836512 2024-06-03T04:57:03.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T04:57:03.346 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672985 2024-06-03T04:57:03.346 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T04:57:03.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T04:57:03.349 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672985 2024-06-03T04:57:03.349 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T04:57:03.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672985 2024-06-03T04:57:03.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-06-03T04:57:03.351 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672985 2024-06-03T04:57:03.351 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T04:57:03.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672985 -lt 42949672985 2024-06-03T04:57:03.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T04:57:03.776 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509462 2024-06-03T04:57:03.776 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T04:57:03.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T04:57:03.779 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509462 2024-06-03T04:57:03.779 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T04:57:03.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509462 2024-06-03T04:57:03.780 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509462 2024-06-03T04:57:03.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-06-03T04:57:03.781 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T04:57:04.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509463 -lt 64424509462 2024-06-03T04:57:04.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T04:57:04.177 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T04:57:04.177 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345932 2024-06-03T04:57:04.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T04:57:04.179 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345932 2024-06-03T04:57:04.179 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T04:57:04.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345932 2024-06-03T04:57:04.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345932' 2024-06-03T04:57:04.181 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 85899345932 2024-06-03T04:57:04.181 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T04:57:04.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345932 -lt 85899345932 2024-06-03T04:57:04.589 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T04:57:04.589 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T04:57:04.589 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T04:57:05.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-06-03T04:57:05.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T04:57:05.100 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T04:57:05.100 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T04:57:05.100 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T04:57:05.100 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T04:57:05.100 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T04:57:05.100 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T04:57:05.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-06-03T04:57:05.534 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T04:57:05.534 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T04:57:05.535 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T04:57:06.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-06-03T04:57:06.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T04:57:06.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T04:57:06.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:261: TEST_0_osd: ceph osd ok-to-stop 0 2024-06-03T04:57:06.443 INFO:tasks.workunit.client.0.smithi088.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-06-03T04:57:06.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:262: TEST_0_osd: ceph osd ok-to-stop 1 2024-06-03T04:57:06.829 INFO:tasks.workunit.client.0.smithi088.stdout:{"ok_to_stop":true,"osds":[1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-06-03T04:57:06.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:263: TEST_0_osd: ceph osd ok-to-stop 2 2024-06-03T04:57:07.232 INFO:tasks.workunit.client.0.smithi088.stdout:{"ok_to_stop":true,"osds":[2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-06-03T04:57:07.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:264: TEST_0_osd: ceph osd ok-to-stop 3 2024-06-03T04:57:07.616 INFO:tasks.workunit.client.0.smithi088.stdout:{"ok_to_stop":true,"osds":[3],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-06-03T04:57:07.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:265: TEST_0_osd: ceph osd ok-to-stop 0 1 2024-06-03T04:57:07.905 INFO:tasks.workunit.client.0.smithi088.stdout:{"ok_to_stop":false,"osds":[0,1],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-06-03T04:57:07.905 INFO:tasks.workunit.client.0.smithi088.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-06-03T04:57:07.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:266: TEST_0_osd: ceph osd ok-to-stop 2 3 2024-06-03T04:57:08.233 INFO:tasks.workunit.client.0.smithi088.stdout:{"ok_to_stop":false,"osds":[2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-06-03T04:57:08.233 INFO:tasks.workunit.client.0.smithi088.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-06-03T04:57:08.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:267: TEST_0_osd: ceph osd ok-to-stop 0 --max 2 2024-06-03T04:57:08.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:267: TEST_0_osd: grep '[0]' 2024-06-03T04:57:08.651 INFO:tasks.workunit.client.0.smithi088.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-06-03T04:57:08.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:268: TEST_0_osd: ceph osd ok-to-stop 1 --max 2 2024-06-03T04:57:08.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:268: TEST_0_osd: grep '[1]' 2024-06-03T04:57:09.037 INFO:tasks.workunit.client.0.smithi088.stdout:{"ok_to_stop":true,"osds":[1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-06-03T04:57:09.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:271: TEST_0_osd: ceph osd pool set ec min_size 2 2024-06-03T04:57:09.929 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 1 min_size to 2 2024-06-03T04:57:09.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:272: TEST_0_osd: wait_for_clean 2024-06-03T04:57:09.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T04:57:09.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T04:57:09.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T04:57:09.949 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T04:57:09.950 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T04:57:09.950 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T04:57:09.950 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T04:57:09.950 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T04:57:09.951 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T04:57:10.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T04:57:10.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T04:57:10.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T04:57:10.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T04:57:10.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T04:57:10.054 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T04:57:10.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T04:57:10.940 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-06-03T04:57:10.940 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-06-03T04:57:10.940 INFO:tasks.workunit.client.0.smithi088.stderr:3' 2024-06-03T04:57:10.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T04:57:10.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T04:57:10.941 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T04:57:11.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836516 2024-06-03T04:57:11.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836516 2024-06-03T04:57:11.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516' 2024-06-03T04:57:11.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T04:57:11.099 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T04:57:11.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672989 2024-06-03T04:57:11.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672989 2024-06-03T04:57:11.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-42949672989' 2024-06-03T04:57:11.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T04:57:11.251 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T04:57:11.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509466 2024-06-03T04:57:11.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509466 2024-06-03T04:57:11.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-42949672989 2-64424509466' 2024-06-03T04:57:11.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T04:57:11.399 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T04:57:11.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345936 2024-06-03T04:57:11.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345936 2024-06-03T04:57:11.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-42949672989 2-64424509466 3-85899345936' 2024-06-03T04:57:11.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T04:57:11.552 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836516 2024-06-03T04:57:11.552 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T04:57:11.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T04:57:11.553 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836516 2024-06-03T04:57:11.553 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T04:57:11.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836516 2024-06-03T04:57:11.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2024-06-03T04:57:11.554 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836516 2024-06-03T04:57:11.554 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T04:57:11.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836514 -lt 21474836516 2024-06-03T04:57:11.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T04:57:12.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T04:57:12.971 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T04:57:13.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836516 -lt 21474836516 2024-06-03T04:57:13.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T04:57:13.391 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672989 2024-06-03T04:57:13.391 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T04:57:13.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T04:57:13.393 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672989 2024-06-03T04:57:13.393 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T04:57:13.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672989 2024-06-03T04:57:13.394 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672989 2024-06-03T04:57:13.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672989' 2024-06-03T04:57:13.394 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T04:57:13.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672989 -lt 42949672989 2024-06-03T04:57:13.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T04:57:13.817 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509466 2024-06-03T04:57:13.817 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T04:57:13.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T04:57:13.819 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509466 2024-06-03T04:57:13.819 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T04:57:13.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509466 2024-06-03T04:57:13.821 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509466 2024-06-03T04:57:13.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509466' 2024-06-03T04:57:13.821 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T04:57:14.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509467 -lt 64424509466 2024-06-03T04:57:14.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T04:57:14.228 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345936 2024-06-03T04:57:14.228 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T04:57:14.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T04:57:14.230 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345936 2024-06-03T04:57:14.230 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T04:57:14.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345936 2024-06-03T04:57:14.231 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 85899345936 2024-06-03T04:57:14.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345936' 2024-06-03T04:57:14.232 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T04:57:14.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345936 -lt 85899345936 2024-06-03T04:57:14.735 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T04:57:14.736 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T04:57:14.736 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T04:57:15.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-06-03T04:57:15.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T04:57:15.294 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T04:57:15.294 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T04:57:15.294 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T04:57:15.294 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T04:57:15.294 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T04:57:15.294 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T04:57:15.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=15 2024-06-03T04:57:15.753 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T04:57:15.753 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T04:57:15.754 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T04:57:16.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 15 = 32 2024-06-03T04:57:16.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 15 '!=' -1 2024-06-03T04:57:16.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-03T04:57:16.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=15 2024-06-03T04:57:16.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T04:57:16.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-03T04:57:16.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T04:57:16.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T04:57:16.391 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T04:57:16.391 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T04:57:16.391 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T04:57:16.392 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T04:57:16.392 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T04:57:16.392 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T04:57:16.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-06-03T04:57:16.810 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T04:57:16.811 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T04:57:16.811 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T04:57:17.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-06-03T04:57:17.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T04:57:17.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T04:57:17.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:274: TEST_0_osd: ceph osd ok-to-stop 0 1 2024-06-03T04:57:17.715 INFO:tasks.workunit.client.0.smithi088.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-06-03T04:57:17.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:275: TEST_0_osd: ceph osd ok-to-stop 2 3 2024-06-03T04:57:18.106 INFO:tasks.workunit.client.0.smithi088.stdout:{"ok_to_stop":true,"osds":[2,3],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-06-03T04:57:18.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:276: TEST_0_osd: ceph osd ok-to-stop 0 1 2 2024-06-03T04:57:18.399 INFO:tasks.workunit.client.0.smithi088.stdout:{"ok_to_stop":false,"osds":[0,1,2],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-06-03T04:57:18.399 INFO:tasks.workunit.client.0.smithi088.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-06-03T04:57:18.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:277: TEST_0_osd: ceph osd ok-to-stop 1 2 3 2024-06-03T04:57:18.680 INFO:tasks.workunit.client.0.smithi088.stdout:{"ok_to_stop":false,"osds":[1,2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-06-03T04:57:18.680 INFO:tasks.workunit.client.0.smithi088.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-06-03T04:57:18.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:279: TEST_0_osd: ceph osd ok-to-stop 0 --max 2 2024-06-03T04:57:18.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:279: TEST_0_osd: grep '[0,1]' 2024-06-03T04:57:19.069 INFO:tasks.workunit.client.0.smithi088.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-06-03T04:57:19.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:280: TEST_0_osd: ceph osd ok-to-stop 0 --max 20 2024-06-03T04:57:19.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:280: TEST_0_osd: grep '[0,1]' 2024-06-03T04:57:19.460 INFO:tasks.workunit.client.0.smithi088.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-06-03T04:57:19.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:281: TEST_0_osd: ceph osd ok-to-stop 2 --max 2 2024-06-03T04:57:19.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:281: TEST_0_osd: grep '[2,3]' 2024-06-03T04:57:19.855 INFO:tasks.workunit.client.0.smithi088.stdout:{"ok_to_stop":true,"osds":[0,2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-06-03T04:57:19.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:282: TEST_0_osd: ceph osd ok-to-stop 2 --max 20 2024-06-03T04:57:19.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:282: TEST_0_osd: grep '[2,3]' 2024-06-03T04:57:20.265 INFO:tasks.workunit.client.0.smithi088.stdout:{"ok_to_stop":true,"osds":[0,2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-06-03T04:57:20.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:285: TEST_0_osd: kill_daemons td/ok-to-stop TERM osd.0 2024-06-03T04:57:20.267 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:57:20.268 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:57:20.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:57:20.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:57:20.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:57:20.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:57:20.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:286: TEST_0_osd: ceph osd down 0 2024-06-03T04:57:21.364 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already down. 2024-06-03T04:57:21.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:287: TEST_0_osd: wait_for_peered 2024-06-03T04:57:21.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: wait_for_peered: local cmd= 2024-06-03T04:57:21.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: wait_for_peered: local num_peered=-1 2024-06-03T04:57:21.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1701: wait_for_peered: local cur_peered 2024-06-03T04:57:21.388 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: get_timeout_delays 90 .1 2024-06-03T04:57:21.389 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T04:57:21.389 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T04:57:21.389 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T04:57:21.389 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T04:57:21.389 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T04:57:21.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T04:57:21.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: local -a delays 2024-06-03T04:57:21.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_peered: local -i loop=0 2024-06-03T04:57:21.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1705: wait_for_peered: flush_pg_stats 2024-06-03T04:57:21.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T04:57:21.507 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T04:57:21.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T04:57:21.952 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-06-03T04:57:21.952 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-06-03T04:57:21.952 INFO:tasks.workunit.client.0.smithi088.stderr:3' 2024-06-03T04:57:21.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T04:57:21.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T04:57:21.953 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T04:57:22.104 INFO:tasks.workunit.client.0.smithi088.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-06-03T04:57:22.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq= 2024-06-03T04:57:22.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z '' 2024-06-03T04:57:22.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: continue 2024-06-03T04:57:22.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T04:57:22.107 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T04:57:22.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672993 2024-06-03T04:57:22.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672993 2024-06-03T04:57:22.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 1-42949672993' 2024-06-03T04:57:22.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T04:57:22.247 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T04:57:22.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509471 2024-06-03T04:57:22.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509471 2024-06-03T04:57:22.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 1-42949672993 2-64424509471' 2024-06-03T04:57:22.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T04:57:22.400 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-03T04:57:22.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345940 2024-06-03T04:57:22.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345940 2024-06-03T04:57:22.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 1-42949672993 2-64424509471 3-85899345940' 2024-06-03T04:57:22.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T04:57:22.538 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672993 2024-06-03T04:57:22.538 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T04:57:22.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T04:57:22.540 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672993 2024-06-03T04:57:22.540 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T04:57:22.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672993 2024-06-03T04:57:22.541 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672993 2024-06-03T04:57:22.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672993' 2024-06-03T04:57:22.541 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T04:57:22.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672993 -lt 42949672993 2024-06-03T04:57:22.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T04:57:22.953 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509471 2024-06-03T04:57:22.953 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T04:57:22.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T04:57:22.954 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509471 2024-06-03T04:57:22.954 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T04:57:22.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509471 2024-06-03T04:57:22.955 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509471 2024-06-03T04:57:22.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509471' 2024-06-03T04:57:22.956 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T04:57:23.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509471 -lt 64424509471 2024-06-03T04:57:23.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T04:57:23.440 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345940 2024-06-03T04:57:23.440 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T04:57:23.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-03T04:57:23.442 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345940 2024-06-03T04:57:23.442 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T04:57:23.443 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345940 2024-06-03T04:57:23.443 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345940' 2024-06-03T04:57:23.443 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 85899345940 2024-06-03T04:57:23.443 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T04:57:23.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345938 -lt 85899345940 2024-06-03T04:57:23.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T04:57:24.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T04:57:24.852 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-03T04:57:25.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345940 -lt 85899345940 2024-06-03T04:57:25.318 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: get_num_pgs 2024-06-03T04:57:25.318 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T04:57:25.318 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T04:57:25.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: test 32 == 0 2024-06-03T04:57:25.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_peered: true 2024-06-03T04:57:25.842 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: get_num_active_or_peered 2024-06-03T04:57:25.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1394: get_num_active_or_peered: local expression 2024-06-03T04:57:25.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2024-06-03T04:57:25.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2024-06-03T04:57:25.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: get_num_active_or_peered: ceph --format json pg dump pgs 2024-06-03T04:57:25.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2024-06-03T04:57:26.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: cur_peered=10 2024-06-03T04:57:26.250 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: get_num_pgs 2024-06-03T04:57:26.250 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T04:57:26.251 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T04:57:26.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: test 10 = 32 2024-06-03T04:57:26.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1716: wait_for_peered: test 10 '!=' -1 2024-06-03T04:57:26.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_peered: loop=0 2024-06-03T04:57:26.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1718: wait_for_peered: num_peered=10 2024-06-03T04:57:26.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1726: wait_for_peered: eval 2024-06-03T04:57:26.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1727: wait_for_peered: sleep 0.1 2024-06-03T04:57:26.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1728: wait_for_peered: loop+=1 2024-06-03T04:57:26.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_peered: true 2024-06-03T04:57:26.916 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: get_num_active_or_peered 2024-06-03T04:57:26.916 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1394: get_num_active_or_peered: local expression 2024-06-03T04:57:26.916 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2024-06-03T04:57:26.916 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2024-06-03T04:57:26.917 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: get_num_active_or_peered: ceph --format json pg dump pgs 2024-06-03T04:57:26.917 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2024-06-03T04:57:27.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: cur_peered=27 2024-06-03T04:57:27.322 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: get_num_pgs 2024-06-03T04:57:27.322 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T04:57:27.322 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T04:57:27.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: test 27 = 32 2024-06-03T04:57:27.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1716: wait_for_peered: test 27 '!=' 10 2024-06-03T04:57:27.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_peered: loop=0 2024-06-03T04:57:27.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1718: wait_for_peered: num_peered=27 2024-06-03T04:57:27.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1726: wait_for_peered: eval 2024-06-03T04:57:27.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1727: wait_for_peered: sleep 0.1 2024-06-03T04:57:27.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1728: wait_for_peered: loop+=1 2024-06-03T04:57:27.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_peered: true 2024-06-03T04:57:27.940 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: get_num_active_or_peered 2024-06-03T04:57:27.940 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1394: get_num_active_or_peered: local expression 2024-06-03T04:57:27.941 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2024-06-03T04:57:27.941 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2024-06-03T04:57:27.941 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: get_num_active_or_peered: ceph --format json pg dump pgs 2024-06-03T04:57:27.941 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2024-06-03T04:57:28.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: cur_peered=27 2024-06-03T04:57:28.375 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: get_num_pgs 2024-06-03T04:57:28.375 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T04:57:28.376 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T04:57:28.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: test 27 = 32 2024-06-03T04:57:28.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1716: wait_for_peered: test 27 '!=' 27 2024-06-03T04:57:28.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_peered: get_is_making_recovery_progress 2024-06-03T04:57:28.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T04:57:28.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T04:57:28.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T04:57:28.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T04:57:28.891 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T04:57:28.891 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T04:57:29.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-03T04:57:29.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-03T04:57:29.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1721: wait_for_peered: (( 1 >= 13 )) 2024-06-03T04:57:29.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1726: wait_for_peered: eval 2024-06-03T04:57:29.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1727: wait_for_peered: sleep 0.2 2024-06-03T04:57:29.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1728: wait_for_peered: loop+=1 2024-06-03T04:57:29.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_peered: true 2024-06-03T04:57:29.671 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: get_num_active_or_peered 2024-06-03T04:57:29.671 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1394: get_num_active_or_peered: local expression 2024-06-03T04:57:29.672 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2024-06-03T04:57:29.672 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2024-06-03T04:57:29.672 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: get_num_active_or_peered: ceph --format json pg dump pgs 2024-06-03T04:57:29.672 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2024-06-03T04:57:30.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: cur_peered=32 2024-06-03T04:57:30.077 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: get_num_pgs 2024-06-03T04:57:30.077 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T04:57:30.077 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T04:57:30.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: test 32 = 32 2024-06-03T04:57:30.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: break 2024-06-03T04:57:30.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1730: wait_for_peered: return 0 2024-06-03T04:57:30.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:289: TEST_0_osd: ceph osd ok-to-stop 0 2024-06-03T04:57:30.975 INFO:tasks.workunit.client.0.smithi088.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":0,"num_not_ok_pgs":0} 2024-06-03T04:57:30.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:290: TEST_0_osd: ceph osd ok-to-stop 0 1 2024-06-03T04:57:31.357 INFO:tasks.workunit.client.0.smithi088.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-06-03T04:57:31.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:291: TEST_0_osd: ceph osd ok-to-stop 0 1 2 2024-06-03T04:57:31.669 INFO:tasks.workunit.client.0.smithi088.stdout:{"ok_to_stop":false,"osds":[0,1,2],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-06-03T04:57:31.669 INFO:tasks.workunit.client.0.smithi088.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-06-03T04:57:31.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:292: TEST_0_osd: ceph osd ok-to-stop 1 2 3 2024-06-03T04:57:31.948 INFO:tasks.workunit.client.0.smithi088.stdout:{"ok_to_stop":false,"osds":[1,2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-06-03T04:57:31.948 INFO:tasks.workunit.client.0.smithi088.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-06-03T04:57:31.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-06-03T04:57:31.954 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:57:31.955 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:57:31.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:57:31.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:57:31.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:57:32.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:57:32.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-06-03T04:57:32.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-06-03T04:57:32.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T04:57:32.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-06-03T04:57:32.067 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:57:32.067 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:57:32.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:57:32.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:57:32.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:57:32.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:57:32.073 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:57:32.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:57:32.074 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:57:32.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:57:32.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:57:32.075 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:57:32.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:57:32.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:57:32.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:57:32.076 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:57:32.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:57:32.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T04:57:32.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-06-03T04:57:32.101 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:57:32.101 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:57:32.102 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:57:32.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47366 2024-06-03T04:57:32.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:57:32.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:57:32.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-06-03T04:57:32.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-06-03T04:57:32.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-06-03T04:57:32.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-06-03T04:57:32.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-06-03T04:57:32.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T04:57:32.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-06-03T04:57:32.104 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:57:32.104 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:57:32.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:57:32.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:57:32.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:57:32.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:57:32.105 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:57:32.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:57:32.106 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:57:32.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:57:32.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:57:32.107 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:57:32.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:57:32.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:57:32.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:57:32.108 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:57:32.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:57:32.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T04:57:32.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-06-03T04:57:32.111 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:57:32.111 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:57:32.111 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:57:32.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47366 2024-06-03T04:57:32.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:57:32.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:57:32.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-06-03T04:57:32.113 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T04:57:32.113 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:57:32.113 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:57:32.113 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.47366 2024-06-03T04:57:32.114 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T04:57:32.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T04:57:32.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T04:57:32.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-06-03T04:57:32.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_1_mon_checks td/ok-to-stop 2024-06-03T04:57:32.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:28: TEST_1_mon_checks: local dir=td/ok-to-stop 2024-06-03T04:57:32.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:30: TEST_1_mon_checks: CEPH_ARGS='--fsid=cfc566bc-c3cf-46dc-a040-e327d99dd7fe --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-06-03T04:57:32.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:32: TEST_1_mon_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-06-03T04:57:32.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-06-03T04:57:32.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T04:57:32.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T04:57:32.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T04:57:32.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-06-03T04:57:32.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-06-03T04:57:32.545 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T04:57:32.545 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:57:32.545 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:57:32.545 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:57:32.546 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:57:32.546 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:57:32.546 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47366/$cluster-$name.asok' 2024-06-03T04:57:32.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47366/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-06-03T04:57:32.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T04:57:32.774 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T04:57:32.774 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:57:32.774 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T04:57:32.774 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T04:57:32.775 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T04:57:32.775 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T04:57:32.775 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T04:57:32.775 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T04:57:32.776 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:57:32.776 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:57:32.776 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:57:32.776 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47366/ceph-mon.a.asok 2024-06-03T04:57:32.776 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:57:32.776 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47366/ceph-mon.a.asok config get fsid 2024-06-03T04:57:32.889 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T04:57:32.889 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:57:32.889 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T04:57:32.889 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T04:57:32.890 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T04:57:32.890 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T04:57:32.890 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T04:57:32.890 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T04:57:32.891 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:57:32.891 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:57:32.891 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:57:32.891 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47366/ceph-mon.a.asok 2024-06-03T04:57:32.891 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:57:32.891 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47366/ceph-mon.a.asok config get mon_host 2024-06-03T04:57:32.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:34: TEST_1_mon_checks: ceph mon ok-to-stop dne 2024-06-03T04:57:33.807 INFO:tasks.workunit.client.0.smithi088.stderr:invalid mon(s) specified: [dne] 2024-06-03T04:57:33.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:35: TEST_1_mon_checks: ceph mon ok-to-stop a 2024-06-03T04:57:33.988 INFO:tasks.workunit.client.0.smithi088.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a] 2024-06-03T04:57:33.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:37: TEST_1_mon_checks: ceph mon ok-to-add-offline 2024-06-03T04:57:34.148 INFO:tasks.workunit.client.0.smithi088.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2024-06-03T04:57:34.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:39: TEST_1_mon_checks: ceph mon ok-to-rm a 2024-06-03T04:57:34.352 INFO:tasks.workunit.client.0.smithi088.stderr:Error EBUSY: removing mon.a would break quorum 2024-06-03T04:57:34.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:40: TEST_1_mon_checks: ceph mon ok-to-rm dne 2024-06-03T04:57:34.596 INFO:tasks.workunit.client.0.smithi088.stderr:mon.dne does not exist 2024-06-03T04:57:34.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-06-03T04:57:34.613 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:57:34.613 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:57:34.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:57:34.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:57:34.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:57:34.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:57:34.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-06-03T04:57:34.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-06-03T04:57:34.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T04:57:34.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-06-03T04:57:34.721 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:57:34.721 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:57:34.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:57:34.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:57:34.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:57:34.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:57:34.726 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:57:34.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:57:34.727 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:57:34.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:57:34.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:57:34.728 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:57:34.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:57:34.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:57:34.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:57:34.730 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:57:34.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:57:34.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T04:57:34.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-06-03T04:57:34.733 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:57:34.733 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:57:34.733 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:57:34.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47366 2024-06-03T04:57:34.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:57:34.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:57:34.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-06-03T04:57:34.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-06-03T04:57:34.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-06-03T04:57:34.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-06-03T04:57:34.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-06-03T04:57:34.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T04:57:34.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-06-03T04:57:34.736 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:57:34.736 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:57:34.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:57:34.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:57:34.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:57:34.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:57:34.737 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:57:34.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:57:34.738 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:57:34.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:57:34.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:57:34.739 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:57:34.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:57:34.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:57:34.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:57:34.740 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:57:34.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:57:34.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T04:57:34.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-06-03T04:57:34.743 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:57:34.744 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:57:34.744 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:57:34.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47366 2024-06-03T04:57:34.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:57:34.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:57:34.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-06-03T04:57:34.746 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T04:57:34.746 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:57:34.746 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:57:34.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.47366 2024-06-03T04:57:34.748 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T04:57:34.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T04:57:34.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T04:57:34.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-06-03T04:57:34.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_2_mons_checks td/ok-to-stop 2024-06-03T04:57:34.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:44: TEST_2_mons_checks: local dir=td/ok-to-stop 2024-06-03T04:57:34.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:46: TEST_2_mons_checks: CEPH_ARGS='--fsid=cfc566bc-c3cf-46dc-a040-e327d99dd7fe --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151 ' 2024-06-03T04:57:34.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:48: TEST_2_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-06-03T04:57:34.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-06-03T04:57:34.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T04:57:34.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T04:57:34.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T04:57:34.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-06-03T04:57:34.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-06-03T04:57:35.154 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T04:57:35.154 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:57:35.154 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:57:35.155 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:57:35.155 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:57:35.155 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:57:35.155 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47366/$cluster-$name.asok' 2024-06-03T04:57:35.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47366/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-06-03T04:57:35.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T04:57:35.358 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T04:57:35.358 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:57:35.358 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T04:57:35.358 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T04:57:35.359 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T04:57:35.359 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T04:57:35.360 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T04:57:35.360 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T04:57:35.360 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:57:35.360 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:57:35.360 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:57:35.360 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47366/ceph-mon.a.asok 2024-06-03T04:57:35.360 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:57:35.360 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47366/ceph-mon.a.asok config get fsid 2024-06-03T04:57:35.469 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T04:57:35.469 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:57:35.469 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T04:57:35.469 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T04:57:35.470 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T04:57:35.471 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T04:57:35.471 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T04:57:35.471 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T04:57:35.471 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:57:35.471 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:57:35.471 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:57:35.471 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47366/ceph-mon.a.asok 2024-06-03T04:57:35.471 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:57:35.471 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47366/ceph-mon.a.asok config get mon_host 2024-06-03T04:57:35.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:49: TEST_2_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-06-03T04:57:35.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-06-03T04:57:35.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T04:57:35.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-06-03T04:57:35.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T04:57:35.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2024-06-03T04:57:35.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-06-03T04:57:35.957 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T04:57:35.958 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:57:35.958 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:57:35.958 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:57:35.958 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:57:35.958 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:57:35.959 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47366/$cluster-$name.asok' 2024-06-03T04:57:35.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47366/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-06-03T04:57:36.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T04:57:36.187 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-06-03T04:57:36.187 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:57:36.187 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-06-03T04:57:36.187 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T04:57:36.188 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T04:57:36.188 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-06-03T04:57:36.188 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-06-03T04:57:36.188 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-06-03T04:57:36.189 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:57:36.189 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:57:36.189 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:57:36.189 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47366/ceph-mon.b.asok 2024-06-03T04:57:36.189 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:57:36.190 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47366/ceph-mon.b.asok config get fsid 2024-06-03T04:57:36.309 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-06-03T04:57:36.309 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:57:36.309 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-06-03T04:57:36.309 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T04:57:36.310 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T04:57:36.311 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-06-03T04:57:36.311 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-06-03T04:57:36.311 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-06-03T04:57:36.311 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:57:36.311 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:57:36.311 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:57:36.311 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47366/ceph-mon.b.asok 2024-06-03T04:57:36.311 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:57:36.311 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47366/ceph-mon.b.asok config get mon_host 2024-06-03T04:57:36.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:51: TEST_2_mons_checks: ceph mon ok-to-stop dne 2024-06-03T04:57:42.637 INFO:tasks.workunit.client.0.smithi088.stderr:invalid mon(s) specified: [dne] 2024-06-03T04:57:42.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:52: TEST_2_mons_checks: ceph mon ok-to-stop a 2024-06-03T04:57:42.921 INFO:tasks.workunit.client.0.smithi088.stderr:Error EBUSY: not enough monitors would be available (b) after stopping mons [a] 2024-06-03T04:57:42.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:53: TEST_2_mons_checks: ceph mon ok-to-stop b 2024-06-03T04:57:43.158 INFO:tasks.workunit.client.0.smithi088.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [b] 2024-06-03T04:57:43.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:54: TEST_2_mons_checks: ceph mon ok-to-stop a b 2024-06-03T04:57:43.348 INFO:tasks.workunit.client.0.smithi088.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a,b] 2024-06-03T04:57:43.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:56: TEST_2_mons_checks: ceph mon ok-to-add-offline 2024-06-03T04:57:43.706 INFO:tasks.workunit.client.0.smithi088.stderr:adding another mon that is not yet online will not break quorum 2024-06-03T04:57:43.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:58: TEST_2_mons_checks: ceph mon ok-to-rm a 2024-06-03T04:57:44.034 INFO:tasks.workunit.client.0.smithi088.stderr:safe to remove mon.a 2024-06-03T04:57:44.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:59: TEST_2_mons_checks: ceph mon ok-to-rm b 2024-06-03T04:57:44.344 INFO:tasks.workunit.client.0.smithi088.stderr:safe to remove mon.b 2024-06-03T04:57:44.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:60: TEST_2_mons_checks: ceph mon ok-to-rm dne 2024-06-03T04:57:44.670 INFO:tasks.workunit.client.0.smithi088.stderr:mon.dne does not exist 2024-06-03T04:57:44.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-06-03T04:57:44.686 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:57:44.686 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:57:44.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:57:44.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:57:44.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:57:44.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:57:44.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-06-03T04:57:44.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-06-03T04:57:44.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T04:57:44.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-06-03T04:57:44.796 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:57:44.796 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:57:44.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:57:44.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:57:44.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:57:44.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:57:44.803 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:57:44.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:57:44.804 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:57:44.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:57:44.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:57:44.806 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:57:44.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:57:44.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:57:44.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:57:44.808 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:57:44.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:57:44.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T04:57:44.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-06-03T04:57:44.812 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:57:44.812 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:57:44.812 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:57:44.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47366 2024-06-03T04:57:44.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:57:44.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:57:44.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-06-03T04:57:44.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-06-03T04:57:44.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-06-03T04:57:44.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-06-03T04:57:44.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-06-03T04:57:44.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T04:57:44.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-06-03T04:57:44.815 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:57:44.815 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:57:44.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:57:44.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:57:44.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:57:44.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:57:44.817 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:57:44.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:57:44.818 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:57:44.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:57:44.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:57:44.819 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:57:44.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:57:44.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:57:44.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:57:44.821 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:57:44.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:57:44.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T04:57:44.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-06-03T04:57:44.824 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:57:44.824 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:57:44.825 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:57:44.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47366 2024-06-03T04:57:44.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:57:44.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:57:44.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-06-03T04:57:44.827 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T04:57:44.827 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:57:44.827 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:57:44.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.47366 2024-06-03T04:57:44.829 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T04:57:44.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T04:57:44.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T04:57:44.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-06-03T04:57:44.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_3_mons_checks td/ok-to-stop 2024-06-03T04:57:44.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:64: TEST_3_mons_checks: local dir=td/ok-to-stop 2024-06-03T04:57:44.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:66: TEST_3_mons_checks: CEPH_ARGS='--fsid=cfc566bc-c3cf-46dc-a040-e327d99dd7fe --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152 ' 2024-06-03T04:57:44.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:68: TEST_3_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-06-03T04:57:44.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-06-03T04:57:44.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T04:57:44.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T04:57:44.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T04:57:44.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-06-03T04:57:44.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-06-03T04:57:45.220 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T04:57:45.220 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:57:45.220 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:57:45.221 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:57:45.221 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:57:45.221 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:57:45.221 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47366/$cluster-$name.asok' 2024-06-03T04:57:45.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47366/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-06-03T04:57:45.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T04:57:45.450 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T04:57:45.450 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:57:45.450 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T04:57:45.450 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T04:57:45.451 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T04:57:45.453 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T04:57:45.453 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T04:57:45.453 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T04:57:45.454 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:57:45.454 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:57:45.454 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:57:45.455 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47366/ceph-mon.a.asok 2024-06-03T04:57:45.455 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:57:45.455 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47366/ceph-mon.a.asok config get fsid 2024-06-03T04:57:45.568 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T04:57:45.568 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:57:45.568 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T04:57:45.568 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T04:57:45.569 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T04:57:45.569 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T04:57:45.569 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T04:57:45.569 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T04:57:45.569 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:57:45.570 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:57:45.570 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:57:45.570 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47366/ceph-mon.a.asok 2024-06-03T04:57:45.570 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:57:45.570 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47366/ceph-mon.a.asok config get mon_host 2024-06-03T04:57:45.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:69: TEST_3_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-06-03T04:57:45.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-06-03T04:57:45.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T04:57:45.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-06-03T04:57:45.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T04:57:45.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2024-06-03T04:57:45.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-06-03T04:57:46.090 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T04:57:46.091 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:57:46.091 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:57:46.091 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:57:46.091 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:57:46.091 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:57:46.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47366/$cluster-$name.asok' 2024-06-03T04:57:46.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47366/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-06-03T04:57:46.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T04:57:46.305 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-06-03T04:57:46.305 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:57:46.305 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-06-03T04:57:46.305 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T04:57:46.306 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T04:57:46.306 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-06-03T04:57:46.306 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-06-03T04:57:46.306 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-06-03T04:57:46.307 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:57:46.307 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:57:46.307 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:57:46.307 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47366/ceph-mon.b.asok 2024-06-03T04:57:46.307 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:57:46.307 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47366/ceph-mon.b.asok config get fsid 2024-06-03T04:57:46.423 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-06-03T04:57:46.423 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:57:46.423 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-06-03T04:57:46.424 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T04:57:46.424 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T04:57:46.425 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-06-03T04:57:46.425 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-06-03T04:57:46.425 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-06-03T04:57:46.425 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:57:46.425 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:57:46.425 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:57:46.425 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47366/ceph-mon.b.asok 2024-06-03T04:57:46.425 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:57:46.426 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47366/ceph-mon.b.asok config get mon_host 2024-06-03T04:57:46.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:70: TEST_3_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2024-06-03T04:57:46.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-06-03T04:57:46.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T04:57:46.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=c 2024-06-03T04:57:46.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T04:57:46.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/c 2024-06-03T04:57:46.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2024-06-03T04:57:46.919 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T04:57:46.919 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:57:46.919 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:57:46.920 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:57:46.920 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:57:46.920 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:57:46.920 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47366/$cluster-$name.asok' 2024-06-03T04:57:46.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47366/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2024-06-03T04:57:47.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T04:57:47.141 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c fsid 2024-06-03T04:57:47.141 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:57:47.141 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-06-03T04:57:47.141 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T04:57:47.142 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T04:57:47.143 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-06-03T04:57:47.143 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-06-03T04:57:47.143 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-06-03T04:57:47.143 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:57:47.143 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:57:47.143 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:57:47.143 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47366/ceph-mon.c.asok 2024-06-03T04:57:47.143 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:57:47.144 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47366/ceph-mon.c.asok config get fsid 2024-06-03T04:57:47.258 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c mon_host 2024-06-03T04:57:47.258 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:57:47.258 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-06-03T04:57:47.258 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T04:57:47.258 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T04:57:47.259 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-06-03T04:57:47.259 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-06-03T04:57:47.259 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-06-03T04:57:47.259 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:57:47.260 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:57:47.260 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:57:47.260 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47366/ceph-mon.c.asok 2024-06-03T04:57:47.260 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:57:47.260 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47366/ceph-mon.c.asok config get mon_host 2024-06-03T04:57:47.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:71: TEST_3_mons_checks: wait_for_quorum 60 3 2024-06-03T04:57:47.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-06-03T04:57:47.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=3 2024-06-03T04:57:47.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-06-03T04:57:47.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 3 ]] 2024-06-03T04:57:47.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-06-03T04:57:47.369 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-06-03T04:57:47.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1717390727 2024-06-03T04:57:47.371 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:57:47.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390667 -lt 1717390727 ]] 2024-06-03T04:57:47.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-03T04:57:47.373 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:57:50.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:57:50.611 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:57:44.853545Z","created":"2024-06-03T04:57:44.853545Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:57:50.612 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:57:44.853545Z","created":"2024-06-03T04:57:44.853545Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:57:50.612 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-03T04:57:50.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-06-03T04:57:50.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-06-03T04:57:50.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-06-03T04:57:50.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-06-03T04:57:50.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-06-03T04:57:50.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:73: TEST_3_mons_checks: ceph mon ok-to-stop dne 2024-06-03T04:57:50.919 INFO:tasks.workunit.client.0.smithi088.stderr:invalid mon(s) specified: [dne] 2024-06-03T04:57:50.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:74: TEST_3_mons_checks: ceph mon ok-to-stop a 2024-06-03T04:57:51.269 INFO:tasks.workunit.client.0.smithi088.stderr:quorum should be preserved (b,c) after stopping [a] 2024-06-03T04:57:51.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:75: TEST_3_mons_checks: ceph mon ok-to-stop b 2024-06-03T04:57:51.623 INFO:tasks.workunit.client.0.smithi088.stderr:quorum should be preserved (a,c) after stopping [b] 2024-06-03T04:57:51.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:76: TEST_3_mons_checks: ceph mon ok-to-stop c 2024-06-03T04:57:51.941 INFO:tasks.workunit.client.0.smithi088.stderr:quorum should be preserved (a,b) after stopping [c] 2024-06-03T04:57:51.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:77: TEST_3_mons_checks: ceph mon ok-to-stop a b 2024-06-03T04:57:52.256 INFO:tasks.workunit.client.0.smithi088.stderr:Error EBUSY: not enough monitors would be available (c) after stopping mons [a,b] 2024-06-03T04:57:52.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:78: TEST_3_mons_checks: ceph mon ok-to-stop b c 2024-06-03T04:57:52.546 INFO:tasks.workunit.client.0.smithi088.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [b,c] 2024-06-03T04:57:52.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:79: TEST_3_mons_checks: ceph mon ok-to-stop a b c 2024-06-03T04:57:52.831 INFO:tasks.workunit.client.0.smithi088.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a,b,c] 2024-06-03T04:57:52.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:81: TEST_3_mons_checks: ceph mon ok-to-add-offline 2024-06-03T04:57:53.105 INFO:tasks.workunit.client.0.smithi088.stderr:adding another mon that is not yet online will not break quorum 2024-06-03T04:57:53.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:83: TEST_3_mons_checks: ceph mon ok-to-rm a 2024-06-03T04:57:53.587 INFO:tasks.workunit.client.0.smithi088.stderr:safe to remove mon.a 2024-06-03T04:57:53.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:84: TEST_3_mons_checks: ceph mon ok-to-rm b 2024-06-03T04:57:53.903 INFO:tasks.workunit.client.0.smithi088.stderr:safe to remove mon.b 2024-06-03T04:57:53.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:85: TEST_3_mons_checks: ceph mon ok-to-rm c 2024-06-03T04:57:54.237 INFO:tasks.workunit.client.0.smithi088.stderr:safe to remove mon.c 2024-06-03T04:57:54.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:87: TEST_3_mons_checks: kill_daemons td/ok-to-stop KILL mon.b 2024-06-03T04:57:54.253 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:57:54.253 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:57:54.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:57:54.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:57:54.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:57:54.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:57:54.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:88: TEST_3_mons_checks: wait_for_quorum 60 2 2024-06-03T04:57:54.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-06-03T04:57:54.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=2 2024-06-03T04:57:54.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-06-03T04:57:54.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 2 ]] 2024-06-03T04:57:54.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-06-03T04:57:54.363 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-06-03T04:57:54.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1717390734 2024-06-03T04:57:54.363 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:57:54.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390674 -lt 1717390734 ]] 2024-06-03T04:57:54.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-06-03T04:57:54.365 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:57:54.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:57:54.589 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:57:44.853545Z","created":"2024-06-03T04:57:44.853545Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:57:54.590 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-06-03T04:57:54.590 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:57:44.853545Z","created":"2024-06-03T04:57:44.853545Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:57:54.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:57:54.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:57:54.623 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:57:54.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390674 -lt 1717390734 ]] 2024-06-03T04:57:54.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-06-03T04:57:54.625 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:57:57.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:57:57.847 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:57:44.853545Z","created":"2024-06-03T04:57:44.853545Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:57:57.847 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:57:44.853545Z","created":"2024-06-03T04:57:44.853545Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:57:57.848 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-06-03T04:57:57.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:57:57.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:57:57.883 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:57:57.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390677 -lt 1717390734 ]] 2024-06-03T04:57:57.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-06-03T04:57:57.883 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:57:58.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:57:58.097 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:57:44.853545Z","created":"2024-06-03T04:57:44.853545Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:57:58.098 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-06-03T04:57:58.098 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:57:44.853545Z","created":"2024-06-03T04:57:44.853545Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:57:58.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:57:58.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:57:58.124 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:57:58.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390678 -lt 1717390734 ]] 2024-06-03T04:57:58.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-06-03T04:57:58.126 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:57:58.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:57:58.332 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:57:44.853545Z","created":"2024-06-03T04:57:44.853545Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:57:58.333 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:57:44.853545Z","created":"2024-06-03T04:57:44.853545Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:57:58.333 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-06-03T04:57:58.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:57:58.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:57:58.366 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:57:58.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390678 -lt 1717390734 ]] 2024-06-03T04:57:58.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-06-03T04:57:58.367 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:57:58.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:57:58.577 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:57:44.853545Z","created":"2024-06-03T04:57:44.853545Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:57:58.578 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:57:44.853545Z","created":"2024-06-03T04:57:44.853545Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:57:58.578 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-06-03T04:57:58.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:57:58.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:57:58.612 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:57:58.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390678 -lt 1717390734 ]] 2024-06-03T04:57:58.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-06-03T04:57:58.613 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:58:01.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:58:01.849 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:57:44.853545Z","created":"2024-06-03T04:57:44.853545Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:01.850 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:57:44.853545Z","created":"2024-06-03T04:57:44.853545Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:01.850 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-06-03T04:58:01.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:58:01.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:58:01.885 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:58:01.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390681 -lt 1717390734 ]] 2024-06-03T04:58:01.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-06-03T04:58:01.886 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:58:02.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:58:02.118 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:57:44.853545Z","created":"2024-06-03T04:57:44.853545Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:02.119 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-06-03T04:58:02.119 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:57:44.853545Z","created":"2024-06-03T04:57:44.853545Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:02.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:58:02.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:58:02.155 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:58:02.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390682 -lt 1717390734 ]] 2024-06-03T04:58:02.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-06-03T04:58:02.156 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:58:02.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:58:02.381 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:57:44.853545Z","created":"2024-06-03T04:57:44.853545Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:02.381 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:57:44.853545Z","created":"2024-06-03T04:57:44.853545Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:02.382 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-06-03T04:58:02.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:58:02.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:58:02.416 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:58:02.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390682 -lt 1717390734 ]] 2024-06-03T04:58:02.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-06-03T04:58:02.418 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:58:02.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:58:02.643 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:57:44.853545Z","created":"2024-06-03T04:57:44.853545Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:02.643 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:57:44.853545Z","created":"2024-06-03T04:57:44.853545Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:02.643 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-06-03T04:58:02.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:58:02.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:58:02.679 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:58:02.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390682 -lt 1717390734 ]] 2024-06-03T04:58:02.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-06-03T04:58:02.681 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:58:02.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:58:02.909 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:57:44.853545Z","created":"2024-06-03T04:57:44.853545Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:02.909 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:57:44.853545Z","created":"2024-06-03T04:57:44.853545Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:02.909 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-06-03T04:58:02.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:58:02.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:58:02.938 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:58:02.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390682 -lt 1717390734 ]] 2024-06-03T04:58:02.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-06-03T04:58:02.940 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:58:03.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:58:03.171 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:57:44.853545Z","created":"2024-06-03T04:57:44.853545Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:03.172 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-06-03T04:58:03.172 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:57:44.853545Z","created":"2024-06-03T04:57:44.853545Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:03.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:58:03.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:58:03.207 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:58:03.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390683 -lt 1717390734 ]] 2024-06-03T04:58:03.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-06-03T04:58:03.209 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:58:03.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:58:03.442 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:57:44.853545Z","created":"2024-06-03T04:57:44.853545Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:03.442 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:57:44.853545Z","created":"2024-06-03T04:57:44.853545Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:03.442 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-06-03T04:58:03.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:58:03.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:58:03.477 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:58:03.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390683 -lt 1717390734 ]] 2024-06-03T04:58:03.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-06-03T04:58:03.477 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:58:03.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:58:03.701 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:57:44.853545Z","created":"2024-06-03T04:57:44.853545Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:03.701 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-06-03T04:58:03.701 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:57:44.853545Z","created":"2024-06-03T04:57:44.853545Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:03.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:58:03.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:58:03.735 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:58:03.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390683 -lt 1717390734 ]] 2024-06-03T04:58:03.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-06-03T04:58:03.737 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:58:03.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:58:03.949 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:57:44.853545Z","created":"2024-06-03T04:57:44.853545Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:03.950 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:57:44.853545Z","created":"2024-06-03T04:57:44.853545Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:03.950 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-06-03T04:58:03.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:58:03.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:58:03.983 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:58:03.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390683 -lt 1717390734 ]] 2024-06-03T04:58:03.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-06-03T04:58:03.984 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:58:04.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:58:04.215 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:57:44.853545Z","created":"2024-06-03T04:57:44.853545Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:04.216 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:57:44.853545Z","created":"2024-06-03T04:57:44.853545Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:04.217 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-06-03T04:58:04.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:58:04.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:58:04.250 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:58:04.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390684 -lt 1717390734 ]] 2024-06-03T04:58:04.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-06-03T04:58:04.251 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:58:10.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:58:10.488 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":8,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:57:44.853545Z","created":"2024-06-03T04:57:44.853545Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:10.488 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-06-03T04:58:10.489 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:57:44.853545Z","created":"2024-06-03T04:57:44.853545Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:10.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-06-03T04:58:10.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-06-03T04:58:10.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-06-03T04:58:10.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-06-03T04:58:10.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-06-03T04:58:10.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:90: TEST_3_mons_checks: ceph mon ok-to-stop a 2024-06-03T04:58:10.733 INFO:tasks.workunit.client.0.smithi088.stderr:Error EBUSY: not enough monitors would be available (c) after stopping mons [a] 2024-06-03T04:58:10.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:91: TEST_3_mons_checks: ceph mon ok-to-stop b 2024-06-03T04:58:11.037 INFO:tasks.workunit.client.0.smithi088.stderr:quorum should be preserved (a,c) after stopping [b] 2024-06-03T04:58:11.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:92: TEST_3_mons_checks: ceph mon ok-to-stop c 2024-06-03T04:58:11.289 INFO:tasks.workunit.client.0.smithi088.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [c] 2024-06-03T04:58:11.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:94: TEST_3_mons_checks: ceph mon ok-to-add-offline 2024-06-03T04:58:11.564 INFO:tasks.workunit.client.0.smithi088.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2024-06-03T04:58:11.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:96: TEST_3_mons_checks: ceph mon ok-to-rm a 2024-06-03T04:58:11.815 INFO:tasks.workunit.client.0.smithi088.stderr:Error EBUSY: removing mon.a would break quorum 2024-06-03T04:58:11.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:97: TEST_3_mons_checks: ceph mon ok-to-rm b 2024-06-03T04:58:12.109 INFO:tasks.workunit.client.0.smithi088.stderr:safe to remove mon.b 2024-06-03T04:58:12.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:98: TEST_3_mons_checks: ceph mon ok-to-rm c 2024-06-03T04:58:12.396 INFO:tasks.workunit.client.0.smithi088.stderr:Error EBUSY: removing mon.c would break quorum 2024-06-03T04:58:12.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-06-03T04:58:12.401 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:58:12.401 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:58:12.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:58:12.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:58:12.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:58:12.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:58:12.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-06-03T04:58:12.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-06-03T04:58:12.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T04:58:12.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-06-03T04:58:12.510 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:58:12.510 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:58:12.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:58:12.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:58:12.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:58:12.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:58:12.517 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:58:12.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:58:12.518 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:58:12.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:58:12.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:58:12.519 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:58:12.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:58:12.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:58:12.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:58:12.521 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:58:12.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:58:12.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T04:58:12.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-06-03T04:58:12.527 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:58:12.527 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:58:12.527 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:58:12.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47366 2024-06-03T04:58:12.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:58:12.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:58:12.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-06-03T04:58:12.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-06-03T04:58:12.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-06-03T04:58:12.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-06-03T04:58:12.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-06-03T04:58:12.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T04:58:12.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-06-03T04:58:12.529 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:58:12.530 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:58:12.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:58:12.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:58:12.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:58:12.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:58:12.531 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:58:12.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:58:12.532 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:58:12.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:58:12.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:58:12.534 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:58:12.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:58:12.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:58:12.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:58:12.535 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:58:12.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:58:12.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T04:58:12.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-06-03T04:58:12.538 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:58:12.538 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:58:12.539 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:58:12.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47366 2024-06-03T04:58:12.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:58:12.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:58:12.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-06-03T04:58:12.541 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T04:58:12.541 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:58:12.541 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:58:12.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.47366 2024-06-03T04:58:12.542 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T04:58:12.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T04:58:12.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T04:58:12.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-06-03T04:58:12.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_4_mons_checks td/ok-to-stop 2024-06-03T04:58:12.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:102: TEST_4_mons_checks: local dir=td/ok-to-stop 2024-06-03T04:58:12.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:104: TEST_4_mons_checks: CEPH_ARGS='--fsid=cfc566bc-c3cf-46dc-a040-e327d99dd7fe --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152,127.0.0.1:7153 ' 2024-06-03T04:58:12.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:106: TEST_4_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-06-03T04:58:12.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-06-03T04:58:12.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T04:58:12.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T04:58:12.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T04:58:12.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-06-03T04:58:12.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-06-03T04:58:12.943 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T04:58:12.943 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:58:12.944 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:58:12.944 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:58:12.944 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:58:12.944 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:58:12.944 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47366/$cluster-$name.asok' 2024-06-03T04:58:12.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47366/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-06-03T04:58:13.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T04:58:13.181 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T04:58:13.181 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:58:13.181 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T04:58:13.181 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T04:58:13.182 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T04:58:13.182 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T04:58:13.182 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T04:58:13.182 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T04:58:13.183 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:58:13.183 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:58:13.183 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:58:13.183 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47366/ceph-mon.a.asok 2024-06-03T04:58:13.183 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:58:13.183 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47366/ceph-mon.a.asok config get fsid 2024-06-03T04:58:13.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T04:58:13.299 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:58:13.299 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T04:58:13.299 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T04:58:13.299 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T04:58:13.299 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T04:58:13.300 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T04:58:13.300 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T04:58:13.300 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:58:13.301 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:58:13.301 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:58:13.301 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47366/ceph-mon.a.asok 2024-06-03T04:58:13.301 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:58:13.301 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47366/ceph-mon.a.asok config get mon_host 2024-06-03T04:58:13.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:107: TEST_4_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-06-03T04:58:13.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-06-03T04:58:13.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T04:58:13.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-06-03T04:58:13.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T04:58:13.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2024-06-03T04:58:13.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-06-03T04:58:13.797 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T04:58:13.797 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:58:13.797 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:58:13.797 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:58:13.797 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:58:13.797 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:58:13.798 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47366/$cluster-$name.asok' 2024-06-03T04:58:13.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47366/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-06-03T04:58:14.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T04:58:14.001 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-06-03T04:58:14.001 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:58:14.001 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-06-03T04:58:14.001 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T04:58:14.002 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T04:58:14.002 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-06-03T04:58:14.002 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-06-03T04:58:14.002 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-06-03T04:58:14.003 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:58:14.003 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:58:14.003 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:58:14.003 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47366/ceph-mon.b.asok 2024-06-03T04:58:14.003 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:58:14.003 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47366/ceph-mon.b.asok config get fsid 2024-06-03T04:58:14.119 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-06-03T04:58:14.120 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:58:14.120 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-06-03T04:58:14.120 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T04:58:14.121 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T04:58:14.121 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-06-03T04:58:14.121 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-06-03T04:58:14.121 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-06-03T04:58:14.121 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:58:14.121 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:58:14.121 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:58:14.121 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47366/ceph-mon.b.asok 2024-06-03T04:58:14.122 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:58:14.122 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47366/ceph-mon.b.asok config get mon_host 2024-06-03T04:58:14.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:108: TEST_4_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2024-06-03T04:58:14.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-06-03T04:58:14.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T04:58:14.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=c 2024-06-03T04:58:14.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T04:58:14.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/c 2024-06-03T04:58:14.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2024-06-03T04:58:14.624 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T04:58:14.624 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:58:14.624 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:58:14.625 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:58:14.625 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:58:14.625 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:58:14.625 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47366/$cluster-$name.asok' 2024-06-03T04:58:14.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47366/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2024-06-03T04:58:14.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T04:58:14.837 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c fsid 2024-06-03T04:58:14.837 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:58:14.837 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-06-03T04:58:14.837 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T04:58:14.838 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T04:58:14.838 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-06-03T04:58:14.838 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-06-03T04:58:14.838 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-06-03T04:58:14.839 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:58:14.839 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:58:14.839 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:58:14.839 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47366/ceph-mon.c.asok 2024-06-03T04:58:14.839 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:58:14.839 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47366/ceph-mon.c.asok config get fsid 2024-06-03T04:58:14.957 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c mon_host 2024-06-03T04:58:14.957 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:58:14.957 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-06-03T04:58:14.957 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T04:58:14.957 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T04:58:14.957 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-06-03T04:58:14.957 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-06-03T04:58:14.958 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-06-03T04:58:14.958 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:58:14.958 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:58:14.958 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:58:14.958 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47366/ceph-mon.c.asok 2024-06-03T04:58:14.958 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:58:14.959 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47366/ceph-mon.c.asok config get mon_host 2024-06-03T04:58:15.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:109: TEST_4_mons_checks: run_mon td/ok-to-stop d --public-addr=127.0.0.1:7153 2024-06-03T04:58:15.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-06-03T04:58:15.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T04:58:15.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=d 2024-06-03T04:58:15.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T04:58:15.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/d 2024-06-03T04:58:15.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id d --mkfs --mon-data=td/ok-to-stop/d --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7153 2024-06-03T04:58:15.470 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T04:58:15.470 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:58:15.470 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:58:15.471 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:58:15.471 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:58:15.471 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:58:15.471 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47366/$cluster-$name.asok' 2024-06-03T04:58:15.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id d --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/d '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47366/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7153 2024-06-03T04:58:15.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T04:58:15.717 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d fsid 2024-06-03T04:58:15.717 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:58:15.718 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2024-06-03T04:58:15.718 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T04:58:15.719 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T04:58:15.719 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2024-06-03T04:58:15.719 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2024-06-03T04:58:15.719 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2024-06-03T04:58:15.719 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:58:15.719 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:58:15.719 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:58:15.720 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47366/ceph-mon.d.asok 2024-06-03T04:58:15.720 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:58:15.720 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47366/ceph-mon.d.asok config get fsid 2024-06-03T04:58:15.836 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d mon_host 2024-06-03T04:58:15.836 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:58:15.836 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2024-06-03T04:58:15.836 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T04:58:15.837 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T04:58:15.837 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2024-06-03T04:58:15.837 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2024-06-03T04:58:15.837 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2024-06-03T04:58:15.838 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:58:15.838 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:58:15.838 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:58:15.838 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47366/ceph-mon.d.asok 2024-06-03T04:58:15.838 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:58:15.838 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47366/ceph-mon.d.asok config get mon_host 2024-06-03T04:58:15.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:110: TEST_4_mons_checks: wait_for_quorum 60 4 2024-06-03T04:58:15.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-06-03T04:58:15.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=4 2024-06-03T04:58:15.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-06-03T04:58:15.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 4 ]] 2024-06-03T04:58:15.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-06-03T04:58:15.948 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-06-03T04:58:15.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1717390755 2024-06-03T04:58:15.949 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:58:15.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390695 -lt 1717390755 ]] 2024-06-03T04:58:15.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-06-03T04:58:15.950 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:58:19.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:58:19.188 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:12.563602Z","created":"2024-06-03T04:58:12.563602Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:19.189 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-06-03T04:58:19.189 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:12.563602Z","created":"2024-06-03T04:58:12.563602Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:19.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-06-03T04:58:19.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-06-03T04:58:19.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-06-03T04:58:19.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-06-03T04:58:19.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-06-03T04:58:19.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:112: TEST_4_mons_checks: ceph mon ok-to-stop dne 2024-06-03T04:58:19.528 INFO:tasks.workunit.client.0.smithi088.stderr:invalid mon(s) specified: [dne] 2024-06-03T04:58:19.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:113: TEST_4_mons_checks: ceph mon ok-to-stop a 2024-06-03T04:58:19.902 INFO:tasks.workunit.client.0.smithi088.stderr:quorum should be preserved (b,c,d) after stopping [a] 2024-06-03T04:58:19.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:114: TEST_4_mons_checks: ceph mon ok-to-stop b 2024-06-03T04:58:20.316 INFO:tasks.workunit.client.0.smithi088.stderr:quorum should be preserved (a,c,d) after stopping [b] 2024-06-03T04:58:20.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:115: TEST_4_mons_checks: ceph mon ok-to-stop c 2024-06-03T04:58:20.700 INFO:tasks.workunit.client.0.smithi088.stderr:quorum should be preserved (a,b,d) after stopping [c] 2024-06-03T04:58:20.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:116: TEST_4_mons_checks: ceph mon ok-to-stop d 2024-06-03T04:58:21.130 INFO:tasks.workunit.client.0.smithi088.stderr:quorum should be preserved (a,b,c) after stopping [d] 2024-06-03T04:58:21.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:117: TEST_4_mons_checks: ceph mon ok-to-stop a b 2024-06-03T04:58:21.452 INFO:tasks.workunit.client.0.smithi088.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [a,b] 2024-06-03T04:58:21.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:118: TEST_4_mons_checks: ceph mon ok-to-stop c d 2024-06-03T04:58:21.755 INFO:tasks.workunit.client.0.smithi088.stderr:Error EBUSY: not enough monitors would be available (a,b) after stopping mons [c,d] 2024-06-03T04:58:21.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:120: TEST_4_mons_checks: ceph mon ok-to-add-offline 2024-06-03T04:58:22.075 INFO:tasks.workunit.client.0.smithi088.stderr:adding another mon that is not yet online will not break quorum 2024-06-03T04:58:22.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:122: TEST_4_mons_checks: ceph mon ok-to-rm a 2024-06-03T04:58:22.399 INFO:tasks.workunit.client.0.smithi088.stderr:safe to remove mon.a 2024-06-03T04:58:22.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:123: TEST_4_mons_checks: ceph mon ok-to-rm b 2024-06-03T04:58:22.789 INFO:tasks.workunit.client.0.smithi088.stderr:safe to remove mon.b 2024-06-03T04:58:22.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:124: TEST_4_mons_checks: ceph mon ok-to-rm c 2024-06-03T04:58:23.183 INFO:tasks.workunit.client.0.smithi088.stderr:safe to remove mon.c 2024-06-03T04:58:23.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:126: TEST_4_mons_checks: kill_daemons td/ok-to-stop KILL mon.a 2024-06-03T04:58:23.198 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:58:23.198 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:58:23.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:58:23.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:58:23.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:58:23.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:58:23.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:127: TEST_4_mons_checks: wait_for_quorum 60 3 2024-06-03T04:58:23.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-06-03T04:58:23.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=3 2024-06-03T04:58:23.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-06-03T04:58:23.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 3 ]] 2024-06-03T04:58:23.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-06-03T04:58:23.308 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-06-03T04:58:23.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1717390763 2024-06-03T04:58:23.309 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:58:23.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390703 -lt 1717390763 ]] 2024-06-03T04:58:23.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-03T04:58:23.310 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:58:23.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:58:23.534 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:12.563602Z","created":"2024-06-03T04:58:12.563602Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:23.535 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-03T04:58:23.535 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:12.563602Z","created":"2024-06-03T04:58:12.563602Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:23.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:58:23.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:58:23.571 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:58:23.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390703 -lt 1717390763 ]] 2024-06-03T04:58:23.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-03T04:58:23.572 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:58:23.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:58:23.803 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:12.563602Z","created":"2024-06-03T04:58:12.563602Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:23.803 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:12.563602Z","created":"2024-06-03T04:58:12.563602Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:23.804 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-03T04:58:23.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:58:23.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:58:23.841 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:58:23.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390703 -lt 1717390763 ]] 2024-06-03T04:58:23.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-03T04:58:23.842 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:58:27.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:58:27.071 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:12.563602Z","created":"2024-06-03T04:58:12.563602Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:27.072 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-03T04:58:27.072 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:12.563602Z","created":"2024-06-03T04:58:12.563602Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:27.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:58:27.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:58:27.107 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:58:27.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390707 -lt 1717390763 ]] 2024-06-03T04:58:27.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-03T04:58:27.108 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:58:27.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:58:27.343 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:12.563602Z","created":"2024-06-03T04:58:12.563602Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:27.343 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-03T04:58:27.343 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:12.563602Z","created":"2024-06-03T04:58:12.563602Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:27.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:58:27.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:58:27.378 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:58:27.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390707 -lt 1717390763 ]] 2024-06-03T04:58:27.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-03T04:58:27.379 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:58:27.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:58:27.609 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:12.563602Z","created":"2024-06-03T04:58:12.563602Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:27.610 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:12.563602Z","created":"2024-06-03T04:58:12.563602Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:27.610 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-03T04:58:27.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:58:27.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:58:27.651 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:58:27.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390707 -lt 1717390763 ]] 2024-06-03T04:58:27.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-03T04:58:27.651 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:58:27.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:58:27.864 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:12.563602Z","created":"2024-06-03T04:58:12.563602Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:27.865 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-03T04:58:27.865 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:12.563602Z","created":"2024-06-03T04:58:12.563602Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:27.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:58:27.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:58:27.898 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:58:27.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390707 -lt 1717390763 ]] 2024-06-03T04:58:27.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-03T04:58:27.899 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:58:28.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:58:28.126 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:12.563602Z","created":"2024-06-03T04:58:12.563602Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:28.127 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:12.563602Z","created":"2024-06-03T04:58:12.563602Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:28.127 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-03T04:58:28.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:58:28.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:58:28.163 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:58:28.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390708 -lt 1717390763 ]] 2024-06-03T04:58:28.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-03T04:58:28.164 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:58:28.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:58:28.383 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:12.563602Z","created":"2024-06-03T04:58:12.563602Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:28.383 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:12.563602Z","created":"2024-06-03T04:58:12.563602Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:28.384 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-03T04:58:28.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:58:28.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:58:28.417 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:58:28.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390708 -lt 1717390763 ]] 2024-06-03T04:58:28.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-03T04:58:28.419 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:58:28.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:58:28.641 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:12.563602Z","created":"2024-06-03T04:58:12.563602Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:28.641 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-03T04:58:28.642 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:12.563602Z","created":"2024-06-03T04:58:12.563602Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:28.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:58:28.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:58:28.674 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:58:28.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390708 -lt 1717390763 ]] 2024-06-03T04:58:28.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-03T04:58:28.676 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:58:28.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:58:28.898 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:12.563602Z","created":"2024-06-03T04:58:12.563602Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:28.898 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-03T04:58:28.898 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:12.563602Z","created":"2024-06-03T04:58:12.563602Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:28.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:58:28.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:58:28.929 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:58:28.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390708 -lt 1717390763 ]] 2024-06-03T04:58:28.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-03T04:58:28.930 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:58:29.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:58:29.149 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:12.563602Z","created":"2024-06-03T04:58:12.563602Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:29.149 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:12.563602Z","created":"2024-06-03T04:58:12.563602Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:29.149 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-03T04:58:29.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:58:29.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:58:29.182 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:58:29.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390709 -lt 1717390763 ]] 2024-06-03T04:58:29.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-03T04:58:29.183 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:58:29.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:58:29.422 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:12.563602Z","created":"2024-06-03T04:58:12.563602Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:29.423 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-03T04:58:29.423 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:12.563602Z","created":"2024-06-03T04:58:12.563602Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:29.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:58:29.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:58:29.457 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:58:29.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390709 -lt 1717390763 ]] 2024-06-03T04:58:29.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-03T04:58:29.459 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:58:29.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:58:29.675 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:12.563602Z","created":"2024-06-03T04:58:12.563602Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:29.676 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-03T04:58:29.677 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:12.563602Z","created":"2024-06-03T04:58:12.563602Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:29.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:58:29.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:58:29.702 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:58:29.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390709 -lt 1717390763 ]] 2024-06-03T04:58:29.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-03T04:58:29.703 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:58:29.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:58:29.930 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:12.563602Z","created":"2024-06-03T04:58:12.563602Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:29.930 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:12.563602Z","created":"2024-06-03T04:58:12.563602Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:29.930 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-03T04:58:29.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:58:29.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:58:29.965 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:58:29.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390709 -lt 1717390763 ]] 2024-06-03T04:58:29.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-03T04:58:29.967 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:58:30.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:58:30.183 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:12.563602Z","created":"2024-06-03T04:58:12.563602Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:30.184 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:12.563602Z","created":"2024-06-03T04:58:12.563602Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:30.185 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-03T04:58:30.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:58:30.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:58:30.217 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:58:30.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390710 -lt 1717390763 ]] 2024-06-03T04:58:30.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-03T04:58:30.218 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:58:39.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:58:39.449 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":6,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:12.563602Z","created":"2024-06-03T04:58:12.563602Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:39.449 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-03T04:58:39.449 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:12.563602Z","created":"2024-06-03T04:58:12.563602Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:58:39.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-06-03T04:58:39.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-06-03T04:58:39.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-06-03T04:58:39.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-06-03T04:58:39.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-06-03T04:58:39.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:129: TEST_4_mons_checks: ceph mon ok-to-stop a 2024-06-03T04:58:42.697 INFO:tasks.workunit.client.0.smithi088.stderr:quorum should be preserved (b,c,d) after stopping [a] 2024-06-03T04:58:42.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:130: TEST_4_mons_checks: ceph mon ok-to-stop b 2024-06-03T04:58:42.976 INFO:tasks.workunit.client.0.smithi088.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [b] 2024-06-03T04:58:42.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:131: TEST_4_mons_checks: ceph mon ok-to-stop c 2024-06-03T04:58:46.180 INFO:tasks.workunit.client.0.smithi088.stderr:Error EBUSY: not enough monitors would be available (b,d) after stopping mons [c] 2024-06-03T04:58:46.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:132: TEST_4_mons_checks: ceph mon ok-to-stop d 2024-06-03T04:58:46.503 INFO:tasks.workunit.client.0.smithi088.stderr:Error EBUSY: not enough monitors would be available (b,c) after stopping mons [d] 2024-06-03T04:58:46.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:134: TEST_4_mons_checks: ceph mon ok-to-add-offline 2024-06-03T04:58:49.736 INFO:tasks.workunit.client.0.smithi088.stderr:adding another mon that is not yet online will not break quorum 2024-06-03T04:58:49.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:136: TEST_4_mons_checks: ceph mon ok-to-rm a 2024-06-03T04:58:52.975 INFO:tasks.workunit.client.0.smithi088.stderr:safe to remove mon.a 2024-06-03T04:58:52.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:137: TEST_4_mons_checks: ceph mon ok-to-rm b 2024-06-03T04:58:53.339 INFO:tasks.workunit.client.0.smithi088.stderr:safe to remove mon.b 2024-06-03T04:58:53.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:138: TEST_4_mons_checks: ceph mon ok-to-rm c 2024-06-03T04:58:53.697 INFO:tasks.workunit.client.0.smithi088.stderr:safe to remove mon.c 2024-06-03T04:58:53.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:139: TEST_4_mons_checks: ceph mon ok-to-rm d 2024-06-03T04:58:56.927 INFO:tasks.workunit.client.0.smithi088.stderr:safe to remove mon.d 2024-06-03T04:58:56.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-06-03T04:58:56.944 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:58:56.944 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:58:56.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:58:56.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:58:56.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:58:57.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:58:57.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-06-03T04:58:57.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-06-03T04:58:57.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T04:58:57.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-06-03T04:58:57.256 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:58:57.256 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:58:57.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:58:57.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:58:57.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:58:57.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:58:57.267 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:58:57.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:58:57.268 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:58:57.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:58:57.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:58:57.269 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:58:57.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:58:57.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:58:57.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:58:57.271 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:58:57.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:58:57.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T04:58:57.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-06-03T04:58:57.279 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:58:57.279 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:58:57.279 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:58:57.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47366 2024-06-03T04:58:57.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:58:57.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:58:57.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-06-03T04:58:57.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-06-03T04:58:57.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-06-03T04:58:57.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-06-03T04:58:57.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-06-03T04:58:57.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T04:58:57.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-06-03T04:58:57.282 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:58:57.282 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:58:57.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:58:57.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:58:57.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:58:57.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:58:57.284 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T04:58:57.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T04:58:57.285 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T04:58:57.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T04:58:57.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T04:58:57.287 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T04:58:57.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:58:57.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T04:58:57.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T04:58:57.288 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T04:58:57.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T04:58:57.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T04:58:57.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-06-03T04:58:57.292 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T04:58:57.292 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:58:57.292 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:58:57.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47366 2024-06-03T04:58:57.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T04:58:57.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T04:58:57.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-06-03T04:58:57.294 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T04:58:57.295 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:58:57.295 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:58:57.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.47366 2024-06-03T04:58:57.297 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T04:58:57.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T04:58:57.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T04:58:57.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-06-03T04:58:57.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_5_mons_checks td/ok-to-stop 2024-06-03T04:58:57.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:143: TEST_5_mons_checks: local dir=td/ok-to-stop 2024-06-03T04:58:57.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:145: TEST_5_mons_checks: CEPH_ARGS='--fsid=cfc566bc-c3cf-46dc-a040-e327d99dd7fe --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152,127.0.0.1:7153,127.0.0.1:7154 ' 2024-06-03T04:58:57.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:147: TEST_5_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-06-03T04:58:57.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-06-03T04:58:57.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T04:58:57.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T04:58:57.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T04:58:57.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-06-03T04:58:57.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-06-03T04:58:57.711 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T04:58:57.711 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:58:57.712 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:58:57.712 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:58:57.712 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:58:57.712 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:58:57.712 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47366/$cluster-$name.asok' 2024-06-03T04:58:57.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47366/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-06-03T04:58:57.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T04:58:57.938 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T04:58:57.938 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:58:57.938 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T04:58:57.938 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T04:58:57.939 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T04:58:57.939 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T04:58:57.939 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T04:58:57.939 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T04:58:57.940 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:58:57.940 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:58:57.940 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:58:57.940 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47366/ceph-mon.a.asok 2024-06-03T04:58:57.940 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:58:57.940 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47366/ceph-mon.a.asok config get fsid 2024-06-03T04:58:58.055 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T04:58:58.055 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:58:58.056 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T04:58:58.056 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T04:58:58.056 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T04:58:58.056 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T04:58:58.056 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T04:58:58.056 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T04:58:58.057 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:58:58.057 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:58:58.057 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:58:58.057 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47366/ceph-mon.a.asok 2024-06-03T04:58:58.058 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:58:58.058 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47366/ceph-mon.a.asok config get mon_host 2024-06-03T04:58:58.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:148: TEST_5_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-06-03T04:58:58.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-06-03T04:58:58.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T04:58:58.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-06-03T04:58:58.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T04:58:58.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2024-06-03T04:58:58.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-06-03T04:58:58.589 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T04:58:58.589 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:58:58.589 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:58:58.590 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:58:58.590 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:58:58.590 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:58:58.590 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47366/$cluster-$name.asok' 2024-06-03T04:58:58.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47366/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-06-03T04:58:58.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T04:58:58.811 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-06-03T04:58:58.811 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:58:58.811 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-06-03T04:58:58.811 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T04:58:58.812 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T04:58:58.812 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-06-03T04:58:58.812 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-06-03T04:58:58.812 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-06-03T04:58:58.812 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:58:58.813 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:58:58.813 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:58:58.813 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47366/ceph-mon.b.asok 2024-06-03T04:58:58.813 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:58:58.813 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47366/ceph-mon.b.asok config get fsid 2024-06-03T04:58:58.926 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-06-03T04:58:58.926 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:58:58.926 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-06-03T04:58:58.927 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T04:58:58.927 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T04:58:58.927 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-06-03T04:58:58.927 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-06-03T04:58:58.927 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-06-03T04:58:58.928 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:58:58.928 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:58:58.928 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:58:58.928 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47366/ceph-mon.b.asok 2024-06-03T04:58:58.928 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:58:58.928 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47366/ceph-mon.b.asok config get mon_host 2024-06-03T04:58:59.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:149: TEST_5_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2024-06-03T04:58:59.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-06-03T04:58:59.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T04:58:59.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=c 2024-06-03T04:58:59.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T04:58:59.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/c 2024-06-03T04:58:59.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2024-06-03T04:58:59.433 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T04:58:59.433 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:58:59.433 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:58:59.434 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:58:59.434 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:58:59.435 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:58:59.435 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47366/$cluster-$name.asok' 2024-06-03T04:58:59.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47366/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2024-06-03T04:58:59.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T04:58:59.674 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c fsid 2024-06-03T04:58:59.674 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:58:59.674 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-06-03T04:58:59.674 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T04:58:59.676 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T04:58:59.676 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-06-03T04:58:59.676 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-06-03T04:58:59.676 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-06-03T04:58:59.676 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:58:59.676 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:58:59.676 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:58:59.676 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47366/ceph-mon.c.asok 2024-06-03T04:58:59.676 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:58:59.676 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47366/ceph-mon.c.asok config get fsid 2024-06-03T04:58:59.791 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c mon_host 2024-06-03T04:58:59.791 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:58:59.791 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-06-03T04:58:59.791 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T04:58:59.792 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T04:58:59.792 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-06-03T04:58:59.792 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-06-03T04:58:59.792 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-06-03T04:58:59.793 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:58:59.793 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:58:59.793 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:58:59.793 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47366/ceph-mon.c.asok 2024-06-03T04:58:59.793 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:58:59.794 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47366/ceph-mon.c.asok config get mon_host 2024-06-03T04:58:59.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:150: TEST_5_mons_checks: run_mon td/ok-to-stop d --public-addr=127.0.0.1:7153 2024-06-03T04:58:59.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-06-03T04:58:59.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T04:58:59.903 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=d 2024-06-03T04:58:59.903 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T04:58:59.903 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/d 2024-06-03T04:58:59.903 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id d --mkfs --mon-data=td/ok-to-stop/d --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7153 2024-06-03T04:59:00.356 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T04:59:00.356 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:59:00.356 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:59:00.356 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:59:00.356 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:59:00.356 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:59:00.357 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47366/$cluster-$name.asok' 2024-06-03T04:59:00.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id d --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/d '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47366/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7153 2024-06-03T04:59:00.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T04:59:00.619 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d fsid 2024-06-03T04:59:00.619 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:59:00.619 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2024-06-03T04:59:00.619 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T04:59:00.619 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T04:59:00.619 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2024-06-03T04:59:00.619 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2024-06-03T04:59:00.619 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2024-06-03T04:59:00.620 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:59:00.620 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:59:00.620 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:59:00.620 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47366/ceph-mon.d.asok 2024-06-03T04:59:00.620 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:59:00.620 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47366/ceph-mon.d.asok config get fsid 2024-06-03T04:59:00.707 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d mon_host 2024-06-03T04:59:00.707 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:59:00.708 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2024-06-03T04:59:00.708 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T04:59:00.709 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T04:59:00.709 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2024-06-03T04:59:00.709 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2024-06-03T04:59:00.709 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2024-06-03T04:59:00.709 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:59:00.709 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:59:00.709 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:59:00.709 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47366/ceph-mon.d.asok 2024-06-03T04:59:00.709 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:59:00.710 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47366/ceph-mon.d.asok config get mon_host 2024-06-03T04:59:00.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:151: TEST_5_mons_checks: run_mon td/ok-to-stop e --public-addr=127.0.0.1:7154 2024-06-03T04:59:00.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-06-03T04:59:00.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T04:59:00.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=e 2024-06-03T04:59:00.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T04:59:00.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/e 2024-06-03T04:59:00.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id e --mkfs --mon-data=td/ok-to-stop/e --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7154 2024-06-03T04:59:01.326 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T04:59:01.326 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T04:59:01.326 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T04:59:01.326 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T04:59:01.327 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:59:01.327 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:59:01.327 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47366/$cluster-$name.asok' 2024-06-03T04:59:01.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id e --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/e '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47366/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7154 2024-06-03T04:59:01.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T04:59:01.548 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon e fsid 2024-06-03T04:59:01.548 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:59:01.548 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=e 2024-06-03T04:59:01.548 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T04:59:01.548 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T04:59:01.549 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.e 2024-06-03T04:59:01.549 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.e 2024-06-03T04:59:01.549 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.e ']' 2024-06-03T04:59:01.550 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:59:01.550 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:59:01.550 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:59:01.550 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47366/ceph-mon.e.asok 2024-06-03T04:59:01.550 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:59:01.550 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47366/ceph-mon.e.asok config get fsid 2024-06-03T04:59:01.666 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon e mon_host 2024-06-03T04:59:01.666 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T04:59:01.666 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=e 2024-06-03T04:59:01.667 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T04:59:01.667 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T04:59:01.667 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.e 2024-06-03T04:59:01.667 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.e 2024-06-03T04:59:01.667 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.e ']' 2024-06-03T04:59:01.667 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T04:59:01.667 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T04:59:01.667 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T04:59:01.667 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47366/ceph-mon.e.asok 2024-06-03T04:59:01.667 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T04:59:01.667 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47366/ceph-mon.e.asok config get mon_host 2024-06-03T04:59:01.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:152: TEST_5_mons_checks: wait_for_quorum 60 5 2024-06-03T04:59:01.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-06-03T04:59:01.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=5 2024-06-03T04:59:01.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-06-03T04:59:01.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 5 ]] 2024-06-03T04:59:01.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-06-03T04:59:01.777 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-06-03T04:59:01.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1717390801 2024-06-03T04:59:01.778 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:59:01.779 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390741 -lt 1717390801 ]] 2024-06-03T04:59:01.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 5' 2024-06-03T04:59:01.780 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:59:06.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:59:06.591 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:06.591 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:06.591 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 5' 2024-06-03T04:59:06.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-06-03T04:59:06.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-06-03T04:59:06.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-06-03T04:59:06.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-06-03T04:59:06.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-06-03T04:59:06.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:154: TEST_5_mons_checks: ceph mon ok-to-stop dne 2024-06-03T04:59:07.004 INFO:tasks.workunit.client.0.smithi088.stderr:invalid mon(s) specified: [dne] 2024-06-03T04:59:07.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:155: TEST_5_mons_checks: ceph mon ok-to-stop a 2024-06-03T04:59:07.404 INFO:tasks.workunit.client.0.smithi088.stderr:quorum should be preserved (b,c,d,e) after stopping [a] 2024-06-03T04:59:07.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:156: TEST_5_mons_checks: ceph mon ok-to-stop b 2024-06-03T04:59:07.770 INFO:tasks.workunit.client.0.smithi088.stderr:quorum should be preserved (a,c,d,e) after stopping [b] 2024-06-03T04:59:07.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:157: TEST_5_mons_checks: ceph mon ok-to-stop c 2024-06-03T04:59:08.225 INFO:tasks.workunit.client.0.smithi088.stderr:quorum should be preserved (a,b,d,e) after stopping [c] 2024-06-03T04:59:08.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:158: TEST_5_mons_checks: ceph mon ok-to-stop d 2024-06-03T04:59:08.634 INFO:tasks.workunit.client.0.smithi088.stderr:quorum should be preserved (a,b,c,e) after stopping [d] 2024-06-03T04:59:08.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:159: TEST_5_mons_checks: ceph mon ok-to-stop e 2024-06-03T04:59:09.033 INFO:tasks.workunit.client.0.smithi088.stderr:quorum should be preserved (a,b,c,d) after stopping [e] 2024-06-03T04:59:09.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:160: TEST_5_mons_checks: ceph mon ok-to-stop a b 2024-06-03T04:59:09.458 INFO:tasks.workunit.client.0.smithi088.stderr:quorum should be preserved (c,d,e) after stopping [a,b] 2024-06-03T04:59:09.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:161: TEST_5_mons_checks: ceph mon ok-to-stop c d 2024-06-03T04:59:09.775 INFO:tasks.workunit.client.0.smithi088.stderr:quorum should be preserved (a,b,e) after stopping [c,d] 2024-06-03T04:59:09.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:162: TEST_5_mons_checks: ceph mon ok-to-stop a b c 2024-06-03T04:59:10.145 INFO:tasks.workunit.client.0.smithi088.stderr:Error EBUSY: not enough monitors would be available (d,e) after stopping mons [a,b,c] 2024-06-03T04:59:10.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:164: TEST_5_mons_checks: ceph mon ok-to-add-offline 2024-06-03T04:59:10.632 INFO:tasks.workunit.client.0.smithi088.stderr:adding another mon that is not yet online will not break quorum 2024-06-03T04:59:10.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:166: TEST_5_mons_checks: ceph mon ok-to-rm a 2024-06-03T04:59:11.050 INFO:tasks.workunit.client.0.smithi088.stderr:safe to remove mon.a 2024-06-03T04:59:11.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:167: TEST_5_mons_checks: ceph mon ok-to-rm b 2024-06-03T04:59:11.427 INFO:tasks.workunit.client.0.smithi088.stderr:safe to remove mon.b 2024-06-03T04:59:11.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:168: TEST_5_mons_checks: ceph mon ok-to-rm c 2024-06-03T04:59:11.772 INFO:tasks.workunit.client.0.smithi088.stderr:safe to remove mon.c 2024-06-03T04:59:11.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:169: TEST_5_mons_checks: ceph mon ok-to-rm d 2024-06-03T04:59:12.175 INFO:tasks.workunit.client.0.smithi088.stderr:safe to remove mon.d 2024-06-03T04:59:12.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:170: TEST_5_mons_checks: ceph mon ok-to-rm e 2024-06-03T04:59:12.575 INFO:tasks.workunit.client.0.smithi088.stderr:safe to remove mon.e 2024-06-03T04:59:12.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:172: TEST_5_mons_checks: kill_daemons td/ok-to-stop KILL mon.a 2024-06-03T04:59:12.591 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:59:12.591 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:59:12.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:59:12.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:59:12.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:59:12.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:59:12.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:173: TEST_5_mons_checks: wait_for_quorum 60 4 2024-06-03T04:59:12.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-06-03T04:59:12.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=4 2024-06-03T04:59:12.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-06-03T04:59:12.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 4 ]] 2024-06-03T04:59:12.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-06-03T04:59:12.701 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-06-03T04:59:12.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1717390812 2024-06-03T04:59:12.703 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:59:12.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390752 -lt 1717390812 ]] 2024-06-03T04:59:12.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-06-03T04:59:12.704 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:59:12.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:59:12.939 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:12.940 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:12.940 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-06-03T04:59:12.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:59:12.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:59:12.975 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:59:12.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390752 -lt 1717390812 ]] 2024-06-03T04:59:12.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-06-03T04:59:12.977 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:59:13.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:59:13.245 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:13.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:13.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-06-03T04:59:13.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:59:13.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:59:13.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:59:13.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390753 -lt 1717390812 ]] 2024-06-03T04:59:13.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-06-03T04:59:13.277 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:59:13.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:59:13.505 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:13.506 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-06-03T04:59:13.506 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:13.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:59:13.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:59:13.533 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:59:13.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390753 -lt 1717390812 ]] 2024-06-03T04:59:13.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-06-03T04:59:13.534 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:59:16.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:59:16.785 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:16.785 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-06-03T04:59:16.786 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:16.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:59:16.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:59:16.823 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:59:16.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390756 -lt 1717390812 ]] 2024-06-03T04:59:16.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-06-03T04:59:16.823 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:59:20.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:59:20.068 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:20.069 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-06-03T04:59:20.069 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:20.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:59:20.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:59:20.105 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:59:20.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390760 -lt 1717390812 ]] 2024-06-03T04:59:20.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-06-03T04:59:20.106 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:59:20.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:59:20.335 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:20.335 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:20.335 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-06-03T04:59:20.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:59:20.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:59:20.365 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:59:20.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390760 -lt 1717390812 ]] 2024-06-03T04:59:20.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-06-03T04:59:20.366 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:59:20.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:59:20.593 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:20.594 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-06-03T04:59:20.594 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:20.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:59:20.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:59:20.621 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:59:20.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390760 -lt 1717390812 ]] 2024-06-03T04:59:20.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-06-03T04:59:20.623 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:59:20.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:59:20.852 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:20.853 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:20.853 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-06-03T04:59:20.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:59:20.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:59:20.888 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:59:20.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390760 -lt 1717390812 ]] 2024-06-03T04:59:20.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-06-03T04:59:20.889 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:59:21.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:59:21.117 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:21.117 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:21.117 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-06-03T04:59:21.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:59:21.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:59:21.150 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:59:21.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390761 -lt 1717390812 ]] 2024-06-03T04:59:21.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-06-03T04:59:21.153 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:59:21.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:59:21.387 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:21.388 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-06-03T04:59:21.388 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:21.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:59:21.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:59:21.422 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:59:21.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390761 -lt 1717390812 ]] 2024-06-03T04:59:21.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-06-03T04:59:21.424 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:59:21.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:59:21.644 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:21.645 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:21.645 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-06-03T04:59:21.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:59:21.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:59:21.679 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:59:21.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390761 -lt 1717390812 ]] 2024-06-03T04:59:21.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-06-03T04:59:21.680 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:59:21.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:59:21.914 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:21.914 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:21.914 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-06-03T04:59:21.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:59:21.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:59:21.949 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:59:21.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390761 -lt 1717390812 ]] 2024-06-03T04:59:21.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-06-03T04:59:21.951 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:59:22.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:59:22.189 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:22.190 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-06-03T04:59:22.190 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:22.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:59:22.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:59:22.225 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:59:22.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390762 -lt 1717390812 ]] 2024-06-03T04:59:22.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-06-03T04:59:22.226 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:59:22.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:59:22.456 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:22.457 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-06-03T04:59:22.457 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:22.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:59:22.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:59:22.487 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:59:22.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390762 -lt 1717390812 ]] 2024-06-03T04:59:22.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-06-03T04:59:22.488 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:59:34.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:59:34.726 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:34.727 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:34.727 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-06-03T04:59:34.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-06-03T04:59:34.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-06-03T04:59:34.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-06-03T04:59:34.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-06-03T04:59:34.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-06-03T04:59:34.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:175: TEST_5_mons_checks: ceph mon ok-to-stop a 2024-06-03T04:59:35.076 INFO:tasks.workunit.client.0.smithi088.stderr:quorum should be preserved (b,c,d,e) after stopping [a] 2024-06-03T04:59:35.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:176: TEST_5_mons_checks: ceph mon ok-to-stop b 2024-06-03T04:59:35.468 INFO:tasks.workunit.client.0.smithi088.stderr:quorum should be preserved (c,d,e) after stopping [b] 2024-06-03T04:59:35.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:177: TEST_5_mons_checks: ceph mon ok-to-stop c 2024-06-03T04:59:35.827 INFO:tasks.workunit.client.0.smithi088.stderr:quorum should be preserved (b,d,e) after stopping [c] 2024-06-03T04:59:35.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:178: TEST_5_mons_checks: ceph mon ok-to-stop d 2024-06-03T04:59:36.174 INFO:tasks.workunit.client.0.smithi088.stderr:quorum should be preserved (b,c,e) after stopping [d] 2024-06-03T04:59:36.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:179: TEST_5_mons_checks: ceph mon ok-to-stop e 2024-06-03T04:59:36.572 INFO:tasks.workunit.client.0.smithi088.stderr:quorum should be preserved (b,c,d) after stopping [e] 2024-06-03T04:59:36.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:181: TEST_5_mons_checks: ceph mon ok-to-add-offline 2024-06-03T04:59:36.948 INFO:tasks.workunit.client.0.smithi088.stderr:adding another mon that is not yet online will not break quorum 2024-06-03T04:59:36.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:183: TEST_5_mons_checks: ceph mon ok-to-rm a 2024-06-03T04:59:37.294 INFO:tasks.workunit.client.0.smithi088.stderr:safe to remove mon.a 2024-06-03T04:59:37.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:184: TEST_5_mons_checks: ceph mon ok-to-rm b 2024-06-03T04:59:40.537 INFO:tasks.workunit.client.0.smithi088.stderr:safe to remove mon.b 2024-06-03T04:59:40.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:185: TEST_5_mons_checks: ceph mon ok-to-rm c 2024-06-03T04:59:40.933 INFO:tasks.workunit.client.0.smithi088.stderr:safe to remove mon.c 2024-06-03T04:59:40.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:186: TEST_5_mons_checks: ceph mon ok-to-rm d 2024-06-03T04:59:41.315 INFO:tasks.workunit.client.0.smithi088.stderr:safe to remove mon.d 2024-06-03T04:59:41.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:187: TEST_5_mons_checks: ceph mon ok-to-rm e 2024-06-03T04:59:41.687 INFO:tasks.workunit.client.0.smithi088.stderr:safe to remove mon.e 2024-06-03T04:59:41.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:189: TEST_5_mons_checks: kill_daemons td/ok-to-stop KILL mon.e 2024-06-03T04:59:41.703 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T04:59:41.703 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T04:59:41.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T04:59:41.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T04:59:41.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T04:59:41.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T04:59:41.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:190: TEST_5_mons_checks: wait_for_quorum 60 3 2024-06-03T04:59:41.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-06-03T04:59:41.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=3 2024-06-03T04:59:41.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-06-03T04:59:41.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 3 ]] 2024-06-03T04:59:41.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-06-03T04:59:41.813 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-06-03T04:59:41.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1717390841 2024-06-03T04:59:41.813 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:59:41.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390781 -lt 1717390841 ]] 2024-06-03T04:59:41.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-03T04:59:41.815 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:59:42.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:59:42.046 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:42.047 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:42.047 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-03T04:59:42.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:59:42.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:59:42.083 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:59:42.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390782 -lt 1717390841 ]] 2024-06-03T04:59:42.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-03T04:59:42.085 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:59:42.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:59:42.310 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:42.310 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-03T04:59:42.311 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:42.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:59:42.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:59:42.341 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:59:42.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390782 -lt 1717390841 ]] 2024-06-03T04:59:42.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-03T04:59:42.343 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:59:42.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:59:42.574 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:42.574 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-03T04:59:42.574 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:42.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:59:42.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:59:42.604 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:59:42.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390782 -lt 1717390841 ]] 2024-06-03T04:59:42.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-03T04:59:42.606 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:59:45.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:59:45.841 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:45.841 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-03T04:59:45.841 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:45.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:59:45.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:59:45.876 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:59:45.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390785 -lt 1717390841 ]] 2024-06-03T04:59:45.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-03T04:59:45.878 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:59:46.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:59:46.114 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:46.118 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-03T04:59:46.118 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:46.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:59:46.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:59:46.146 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:59:46.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390786 -lt 1717390841 ]] 2024-06-03T04:59:46.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-03T04:59:46.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:59:46.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:59:46.380 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:46.381 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:46.381 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-03T04:59:46.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:59:46.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:59:46.415 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:59:46.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390786 -lt 1717390841 ]] 2024-06-03T04:59:46.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-03T04:59:46.417 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:59:46.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:59:46.658 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:46.658 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:46.659 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-03T04:59:46.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:59:46.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:59:46.670 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:59:46.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390786 -lt 1717390841 ]] 2024-06-03T04:59:46.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-03T04:59:46.670 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:59:49.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:59:49.900 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:49.900 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:49.901 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-03T04:59:49.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-03T04:59:49.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-03T04:59:49.935 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-03T04:59:49.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1717390789 -lt 1717390841 ]] 2024-06-03T04:59:49.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-03T04:59:49.936 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-03T04:59:57.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-03T04:59:57.756 INFO:tasks.workunit.client.0.smithi088.stderr:{"election_epoch":14,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:57.757 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-03T04:59:57.757 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":14,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cfc566bc-c3cf-46dc-a040-e327d99dd7fe","modified":"2024-06-03T04:58:57.321578Z","created":"2024-06-03T04:58:57.321578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-03T04:59:57.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-06-03T04:59:57.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-06-03T04:59:57.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-06-03T04:59:57.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-06-03T04:59:57.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-06-03T04:59:57.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:192: TEST_5_mons_checks: ceph mon ok-to-stop a 2024-06-03T05:00:01.022 INFO:tasks.workunit.client.0.smithi088.stderr:quorum should be preserved (b,c,d) after stopping [a] 2024-06-03T05:00:01.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:193: TEST_5_mons_checks: ceph mon ok-to-stop b 2024-06-03T05:00:01.348 INFO:tasks.workunit.client.0.smithi088.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [b] 2024-06-03T05:00:01.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:194: TEST_5_mons_checks: ceph mon ok-to-stop c 2024-06-03T05:00:01.642 INFO:tasks.workunit.client.0.smithi088.stderr:Error EBUSY: not enough monitors would be available (b,d) after stopping mons [c] 2024-06-03T05:00:01.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:195: TEST_5_mons_checks: ceph mon ok-to-stop d 2024-06-03T05:00:04.832 INFO:tasks.workunit.client.0.smithi088.stderr:Error EBUSY: not enough monitors would be available (b,c) after stopping mons [d] 2024-06-03T05:00:04.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:196: TEST_5_mons_checks: ceph mon ok-to-stop e 2024-06-03T05:00:05.144 INFO:tasks.workunit.client.0.smithi088.stderr:quorum should be preserved (b,c,d) after stopping [e] 2024-06-03T05:00:05.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:198: TEST_5_mons_checks: ceph mon ok-to-add-offline 2024-06-03T05:00:05.491 INFO:tasks.workunit.client.0.smithi088.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2024-06-03T05:00:05.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:200: TEST_5_mons_checks: ceph mon ok-to-rm a 2024-06-03T05:00:05.815 INFO:tasks.workunit.client.0.smithi088.stderr:safe to remove mon.a 2024-06-03T05:00:05.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:201: TEST_5_mons_checks: ceph mon ok-to-rm b 2024-06-03T05:00:09.015 INFO:tasks.workunit.client.0.smithi088.stderr:Error EBUSY: removing mon.b would break quorum 2024-06-03T05:00:09.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:202: TEST_5_mons_checks: ceph mon ok-to-rm c 2024-06-03T05:00:09.197 INFO:tasks.workunit.client.0.smithi088.stderr:Error EBUSY: removing mon.c would break quorum 2024-06-03T05:00:09.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:203: TEST_5_mons_checks: ceph mon ok-to-rm d 2024-06-03T05:00:09.398 INFO:tasks.workunit.client.0.smithi088.stderr:Error EBUSY: removing mon.d would break quorum 2024-06-03T05:00:09.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:204: TEST_5_mons_checks: ceph mon ok-to-rm e 2024-06-03T05:00:12.621 INFO:tasks.workunit.client.0.smithi088.stderr:safe to remove mon.e 2024-06-03T05:00:12.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-06-03T05:00:12.637 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:00:12.637 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:00:12.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:00:12.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:00:12.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:00:12.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:00:12.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-06-03T05:00:12.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-06-03T05:00:12.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:00:12.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-06-03T05:00:12.747 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:00:12.747 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:00:12.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:00:12.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:00:12.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:00:12.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:00:12.756 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:00:12.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:00:12.756 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:00:12.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:00:12.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:00:12.757 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:00:12.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:00:12.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:00:12.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:00:12.759 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:00:12.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:00:12.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:00:12.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-06-03T05:00:12.770 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:00:12.770 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:00:12.770 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T05:00:12.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47366 2024-06-03T05:00:12.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:00:12.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:00:12.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-06-03T05:00:12.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/ok-to-stop 0 2024-06-03T05:00:12.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-06-03T05:00:12.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-06-03T05:00:12.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-06-03T05:00:12.772 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:00:12.772 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:00:12.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:00:12.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:00:12.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:00:12.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:00:12.773 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:00:12.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:00:12.774 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:00:12.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:00:12.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:00:12.776 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:00:12.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:00:12.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:00:12.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:00:12.777 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:00:12.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:00:12.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-06-03T05:00:12.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-06-03T05:00:12.779 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:00:12.779 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:00:12.780 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47366 2024-06-03T05:00:12.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47366 2024-06-03T05:00:12.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:00:12.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:00:12.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-06-03T05:00:12.781 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-06-03T05:00:12.782 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-06-03T05:00:12.836 INFO:tasks.workunit:Running workunit misc/rados-striper.sh... 2024-06-03T05:00:12.836 DEBUG:teuthology.orchestra.run.smithi088:workunit test misc/rados-striper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dfcc4b3c2c55532c4c04fa47551c9df1dffdf746 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh 2024-06-03T05:00:12.887 INFO:tasks.workunit.client.0.smithi088.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-06-03T05:00:12.892 INFO:tasks.workunit.client.0.smithi088.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-06-03T05:00:12.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-03T05:00:12.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-03T05:00:12.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-06-03T05:00:12.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-06-03T05:00:12.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-06-03T05:00:12.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-06-03T05:00:12.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-06-03T05:00:12.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-06-03T05:00:12.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/rados-striper 2024-06-03T05:00:12.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:20: run: local dir=td/rados-striper 2024-06-03T05:00:12.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:21: run: shift 2024-06-03T05:00:12.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:23: run: export CEPH_MON=127.0.0.1:7116 2024-06-03T05:00:12.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:23: run: CEPH_MON=127.0.0.1:7116 2024-06-03T05:00:12.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:24: run: export CEPH_ARGS 2024-06-03T05:00:12.893 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:25: run: uuidgen 2024-06-03T05:00:12.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:25: run: CEPH_ARGS+='--fsid=8960adb3-19d6-4e8f-aca7-e5a821f5594c --auth-supported=none ' 2024-06-03T05:00:12.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7116 ' 2024-06-03T05:00:12.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:29: run: setup td/rados-striper 2024-06-03T05:00:12.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/rados-striper 2024-06-03T05:00:12.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/rados-striper 2024-06-03T05:00:12.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/rados-striper 2024-06-03T05:00:12.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:00:12.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/rados-striper KILL 2024-06-03T05:00:12.896 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:00:12.896 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:00:12.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:00:12.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:00:12.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:00:12.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:00:12.897 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:00:12.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:00:12.897 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:00:12.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:00:12.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:00:12.897 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:00:12.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:00:12.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:00:12.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:00:12.898 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:00:12.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:00:12.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:00:12.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/rados-striper 2024-06-03T05:00:12.901 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:00:12.901 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:00:12.901 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64743 2024-06-03T05:00:12.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.64743 2024-06-03T05:00:12.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:00:12.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:00:12.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/rados-striper 2024-06-03T05:00:12.904 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:00:12.904 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:00:12.904 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64743 2024-06-03T05:00:12.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.64743 2024-06-03T05:00:12.906 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:00:12.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-06-03T05:00:12.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-06-03T05:00:12.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:00:12.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/rados-striper 1' TERM HUP INT 2024-06-03T05:00:12.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:32: run: run_mon td/rados-striper a 2024-06-03T05:00:12.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/rados-striper 2024-06-03T05:00:12.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:00:12.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:00:12.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:00:12.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/rados-striper/a 2024-06-03T05:00:12.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/rados-striper/a --run-dir=td/rados-striper 2024-06-03T05:00:13.305 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:00:13.305 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:00:13.305 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:00:13.306 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:00:13.306 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:00:13.306 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64743 2024-06-03T05:00:13.306 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.64743/$cluster-$name.asok' 2024-06-03T05:00:13.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/rados-striper/a '--log-file=td/rados-striper/$name.log' '--admin-socket=/tmp/ceph-asok.64743/$cluster-$name.asok' --mon-cluster-log-file=td/rados-striper/log --run-dir=td/rados-striper '--pid-file=td/rados-striper/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-03T05:00:13.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:00:13.524 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:00:13.524 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:00:13.524 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:00:13.524 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:00:13.525 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:00:13.525 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:00:13.525 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:00:13.525 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:00:13.525 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:00:13.526 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:00:13.526 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64743 2024-06-03T05:00:13.526 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.64743/ceph-mon.a.asok 2024-06-03T05:00:13.526 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:00:13.526 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.64743/ceph-mon.a.asok config get fsid 2024-06-03T05:00:13.641 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:00:13.641 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:00:13.641 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:00:13.641 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:00:13.642 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:00:13.643 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:00:13.643 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:00:13.643 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:00:13.643 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:00:13.643 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:00:13.643 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64743 2024-06-03T05:00:13.643 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.64743/ceph-mon.a.asok 2024-06-03T05:00:13.643 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:00:13.644 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.64743/ceph-mon.a.asok config get mon_host 2024-06-03T05:00:13.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:33: run: run_osd td/rados-striper 0 2024-06-03T05:00:13.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/rados-striper 2024-06-03T05:00:13.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:00:13.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T05:00:13.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:00:13.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/rados-striper/0 2024-06-03T05:00:13.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8960adb3-19d6-4e8f-aca7-e5a821f5594c --auth-supported=none --mon-host=127.0.0.1:7116 ' 2024-06-03T05:00:13.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:00:13.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:00:13.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:00:13.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/rados-striper/0' 2024-06-03T05:00:13.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/rados-striper/0/journal' 2024-06-03T05:00:13.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:00:13.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:00:13.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2024-06-03T05:00:13.754 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:00:13.754 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:00:13.754 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:00:13.755 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:00:13.756 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:00:13.756 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64743 2024-06-03T05:00:13.756 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.64743/$cluster-$name.asok' 2024-06-03T05:00:13.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.64743/$cluster-$name.asok' 2024-06-03T05:00:13.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:00:13.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:00:13.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:00:13.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2024-06-03T05:00:13.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2024-06-03T05:00:13.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:00:13.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:00:13.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:00:13.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:00:13.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:00:13.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:00:13.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/rados-striper/0 2024-06-03T05:00:13.757 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:00:13.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=696bcaa0-6e04-4409-adce-9a265c2cb3b0 2024-06-03T05:00:13.758 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 696bcaa0-6e04-4409-adce-9a265c2cb3b0 2024-06-03T05:00:13.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 696bcaa0-6e04-4409-adce-9a265c2cb3b0' 2024-06-03T05:00:13.758 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:00:13.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDdTV1mCohCLhAA69HxcponsvxAZftwK5MAvA== 2024-06-03T05:00:13.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDdTV1mCohCLhAA69HxcponsvxAZftwK5MAvA=="}' 2024-06-03T05:00:13.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 696bcaa0-6e04-4409-adce-9a265c2cb3b0 -i td/rados-striper/0/new.json 2024-06-03T05:00:14.670 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:00:14.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/rados-striper/0/new.json 2024-06-03T05:00:14.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8960adb3-19d6-4e8f-aca7-e5a821f5594c --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/0 --osd-journal=td/rados-striper/0/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.64743/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDdTV1mCohCLhAA69HxcponsvxAZftwK5MAvA== --osd-uuid 696bcaa0-6e04-4409-adce-9a265c2cb3b0 2024-06-03T05:00:14.719 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:00:14.715+0000 7f4d943a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:00:14.748 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:00:14.743+0000 7f4d943a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:00:14.749 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:00:14.747+0000 7f4d943a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:00:14.750 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:00:14.747+0000 7f4d943a27c0 -1 bluestore(td/rados-striper/0/block) _read_bdev_label failed to open td/rados-striper/0/block: (2) No such file or directory 2024-06-03T05:00:14.750 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:00:14.747+0000 7f4d943a27c0 -1 bluestore(td/rados-striper/0/block) _read_bdev_label failed to open td/rados-striper/0/block: (2) No such file or directory 2024-06-03T05:00:14.750 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:00:14.747+0000 7f4d943a27c0 -1 bluestore(td/rados-striper/0/block) _read_bdev_label failed to open td/rados-striper/0/block: (2) No such file or directory 2024-06-03T05:00:14.779 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:00:14.775+0000 7f4d943a27c0 -1 bluestore(td/rados-striper/0) _read_fsid unparsable uuid 2024-06-03T05:00:21.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/rados-striper/0/keyring 2024-06-03T05:00:21.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:00:21.144 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T05:00:21.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T05:00:21.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/rados-striper/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:00:21.383 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:00:21.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T05:00:21.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:00:21.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:00:21.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8960adb3-19d6-4e8f-aca7-e5a821f5594c --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/0 --osd-journal=td/rados-striper/0/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.64743/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:00:21.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:00:21.413 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:00:21.407+0000 7fdcbde677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:00:21.422 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:00:21.419+0000 7fdcbde677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:00:21.423 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:00:21.419+0000 7fdcbde677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:00:21.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T05:00:21.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:00:21.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:00:21.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:00:21.620 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:00:21.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:00:21.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:00:21.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:00:21.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:00:21.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:00:21.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:00:21.984 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:00:21.979+0000 7fdcbde677c0 -1 Falling back to public interface 2024-06-03T05:00:22.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:00:22.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:00:22.846 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:00:22.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:00:22.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:00:22.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:00:23.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:00:24.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:00:24.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:00:24.066 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:00:24.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:00:24.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:00:24.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:00:24.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:00:24.687 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:00:24.683+0000 7fdcbde677c0 -1 osd.0 0 log_to_monitors true 2024-06-03T05:00:25.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:00:25.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:00:25.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:00:25.305 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:00:25.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:00:25.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:00:25.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:00:26.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:00:26.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:00:26.613 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:00:26.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:00:26.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:00:26.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:00:26.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:00:27.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:00:27.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:00:27.848 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:00:27.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:00:27.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:00:27.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:00:28.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:00:29.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:00:29.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:00:29.082 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:00:29.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:00:29.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:00:29.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:00:29.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:00:30.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:00:30.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:00:30.315 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:00:30.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:00:30.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:00:30.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:00:30.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:00:31.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:00:31.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:00:31.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:00:31.543 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:00:31.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:00:31.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:00:31.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:00:32.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:00:32.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:00:32.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:00:32.775 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:00:32.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:00:32.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:00:33.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:00:34.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:00:34.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:00:34.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:00:34.008 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:00:34.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:00:34.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:00:34.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:00:35.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:00:35.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:00:35.238 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:00:35.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:00:35.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:00:35.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:00:35.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:00:36.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:00:36.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:00:36.460 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:00:36.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:00:36.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:00:36.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:00:36.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:00:37.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:00:37.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:00:37.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:00:37.689 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:00:37.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:00:37.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:00:37.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:00:38.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:00:38.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:00:38.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:00:38.920 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:00:38.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:00:38.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:00:39.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:00:40.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:00:40.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:00:40.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:00:40.152 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:00:40.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:00:40.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:00:40.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:00:41.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:00:41.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:00:41.385 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:00:41.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:00:41.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:00:41.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:00:41.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:00:42.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:00:42.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:00:42.614 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:00:42.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:00:42.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:00:42.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:00:42.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:00:43.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:00:43.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:00:43.845 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-06-03T05:00:43.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T05:00:43.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:00:43.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:00:44.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:00:45.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:00:45.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:00:45.081 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-06-03T05:00:45.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-03T05:00:45.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:00:45.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:00:45.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:00:46.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:00:46.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:00:46.315 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-06-03T05:00:46.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-03T05:00:46.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:00:46.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:00:46.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:00:47.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:00:47.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:00:47.553 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-06-03T05:00:47.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-03T05:00:47.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:00:47.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:00:47.778 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2939668687,v1:127.0.0.1:6801/2939668687] [v2:127.0.0.1:6802/2939668687,v1:127.0.0.1:6803/2939668687] exists,up 696bcaa0-6e04-4409-adce-9a265c2cb3b0 2024-06-03T05:00:47.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:00:47.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:00:47.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:00:47.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:34: run: run_osd td/rados-striper 1 2024-06-03T05:00:47.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/rados-striper 2024-06-03T05:00:47.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:00:47.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-03T05:00:47.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:00:47.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/rados-striper/1 2024-06-03T05:00:47.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8960adb3-19d6-4e8f-aca7-e5a821f5594c --auth-supported=none --mon-host=127.0.0.1:7116 ' 2024-06-03T05:00:47.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:00:47.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:00:47.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:00:47.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/rados-striper/1' 2024-06-03T05:00:47.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/rados-striper/1/journal' 2024-06-03T05:00:47.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:00:47.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:00:47.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2024-06-03T05:00:47.783 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:00:47.783 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:00:47.784 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:00:47.784 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:00:47.784 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:00:47.784 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64743 2024-06-03T05:00:47.784 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.64743/$cluster-$name.asok' 2024-06-03T05:00:47.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.64743/$cluster-$name.asok' 2024-06-03T05:00:47.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:00:47.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:00:47.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:00:47.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2024-06-03T05:00:47.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2024-06-03T05:00:47.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:00:47.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:00:47.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:00:47.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:00:47.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:00:47.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:00:47.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/rados-striper/1 2024-06-03T05:00:47.785 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:00:47.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f6f71f75-083d-4ee0-857c-89b9f95a7f4f 2024-06-03T05:00:47.786 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 f6f71f75-083d-4ee0-857c-89b9f95a7f4f 2024-06-03T05:00:47.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f6f71f75-083d-4ee0-857c-89b9f95a7f4f' 2024-06-03T05:00:47.786 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:00:47.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD/TV1mAEjmLxAAtCFhinTuABdJS+YmEyY6GA== 2024-06-03T05:00:47.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD/TV1mAEjmLxAAtCFhinTuABdJS+YmEyY6GA=="}' 2024-06-03T05:00:47.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f6f71f75-083d-4ee0-857c-89b9f95a7f4f -i td/rados-striper/1/new.json 2024-06-03T05:00:48.192 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:00:48.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/rados-striper/1/new.json 2024-06-03T05:00:48.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=8960adb3-19d6-4e8f-aca7-e5a821f5594c --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/1 --osd-journal=td/rados-striper/1/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.64743/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD/TV1mAEjmLxAAtCFhinTuABdJS+YmEyY6GA== --osd-uuid f6f71f75-083d-4ee0-857c-89b9f95a7f4f 2024-06-03T05:00:48.244 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:00:48.239+0000 7f717d0227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:00:48.286 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:00:48.283+0000 7f717d0227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:00:48.287 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:00:48.283+0000 7f717d0227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:00:48.288 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:00:48.283+0000 7f717d0227c0 -1 bluestore(td/rados-striper/1/block) _read_bdev_label failed to open td/rados-striper/1/block: (2) No such file or directory 2024-06-03T05:00:48.288 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:00:48.283+0000 7f717d0227c0 -1 bluestore(td/rados-striper/1/block) _read_bdev_label failed to open td/rados-striper/1/block: (2) No such file or directory 2024-06-03T05:00:48.288 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:00:48.283+0000 7f717d0227c0 -1 bluestore(td/rados-striper/1/block) _read_bdev_label failed to open td/rados-striper/1/block: (2) No such file or directory 2024-06-03T05:00:48.326 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:00:48.323+0000 7f717d0227c0 -1 bluestore(td/rados-striper/1) _read_fsid unparsable uuid 2024-06-03T05:00:54.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/rados-striper/1/keyring 2024-06-03T05:00:54.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:00:54.845 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-06-03T05:00:54.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-03T05:00:54.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/rados-striper/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:00:55.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-03T05:00:55.088 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-06-03T05:00:55.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=8960adb3-19d6-4e8f-aca7-e5a821f5594c --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/1 --osd-journal=td/rados-striper/1/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.64743/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:00:55.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:00:55.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:00:55.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:00:55.115 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:00:55.111+0000 7f31b564a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:00:55.117 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:00:55.111+0000 7f31b564a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:00:55.118 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:00:55.115+0000 7f31b564a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:00:55.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-03T05:00:55.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:00:55.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T05:00:55.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:00:55.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:00:55.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:00:55.309 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:00:55.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:00:55.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:00:55.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:00:55.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:00:55.680 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:00:55.675+0000 7f31b564a7c0 -1 Falling back to public interface 2024-06-03T05:00:56.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:00:56.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:00:56.535 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:00:56.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:00:56.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:00:56.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:00:56.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:00:57.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:00:57.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:00:57.766 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:00:57.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:00:57.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:00:57.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:00:57.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:00:58.075 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:00:58.071+0000 7f31b564a7c0 -1 osd.1 0 log_to_monitors true 2024-06-03T05:00:59.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:00:59.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:00:59.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:00:59.001 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:00:59.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:00:59.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:00:59.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:01:00.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:01:00.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:01:00.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:01:00.355 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:01:00.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:01:00.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:01:00.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:01:01.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:01:01.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:01:01.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:01:01.587 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:01:01.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:01:01.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:01:01.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:01:02.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:01:02.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:01:02.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:01:02.807 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:01:02.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:01:02.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:01:03.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:01:04.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:01:04.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:01:04.039 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:01:04.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:01:04.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:01:04.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:01:04.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:01:05.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:01:05.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:01:05.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:01:05.271 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:01:05.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:01:05.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:01:05.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:01:06.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:01:06.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:01:06.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:01:06.504 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:01:06.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:01:06.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:01:06.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:01:07.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:01:07.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:01:07.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:01:07.738 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:01:07.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:01:07.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:01:07.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:01:08.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:01:08.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:01:08.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:01:08.966 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:01:08.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:01:08.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:01:09.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:01:10.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:01:10.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:01:10.202 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:01:10.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:01:10.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:01:10.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:01:10.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:01:11.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:01:11.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:01:11.436 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:01:11.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:01:11.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:01:11.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:01:11.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:01:12.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:01:12.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:01:12.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:01:12.667 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:01:12.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:01:12.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:01:12.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:01:13.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:01:13.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:01:13.891 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:01:13.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:01:13.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:01:13.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:01:14.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:01:15.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:01:15.127 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:01:15.127 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:01:15.127 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:01:15.127 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:01:15.127 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:01:15.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:01:16.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:01:16.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:01:16.359 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:01:16.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:01:16.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:01:16.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:01:16.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:01:17.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:01:17.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:01:17.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T05:01:17.594 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-06-03T05:01:17.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:01:17.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:01:17.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:01:18.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:01:18.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:01:18.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-03T05:01:18.826 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-06-03T05:01:18.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:01:18.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:01:19.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:01:20.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:01:20.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:01:20.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-03T05:01:20.052 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-06-03T05:01:20.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:01:20.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:01:20.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:01:21.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:01:21.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:01:21.277 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-06-03T05:01:21.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-03T05:01:21.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:01:21.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:01:21.544 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/3954526062,v1:127.0.0.1:6809/3954526062] [v2:127.0.0.1:6810/3954526062,v1:127.0.0.1:6811/3954526062] exists,up f6f71f75-083d-4ee0-857c-89b9f95a7f4f 2024-06-03T05:01:21.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:01:21.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:01:21.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:01:21.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:35: run: run_osd td/rados-striper 2 2024-06-03T05:01:21.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/rados-striper 2024-06-03T05:01:21.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:01:21.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-03T05:01:21.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:01:21.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/rados-striper/2 2024-06-03T05:01:21.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8960adb3-19d6-4e8f-aca7-e5a821f5594c --auth-supported=none --mon-host=127.0.0.1:7116 ' 2024-06-03T05:01:21.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:01:21.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:01:21.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:01:21.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/rados-striper/2' 2024-06-03T05:01:21.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/rados-striper/2/journal' 2024-06-03T05:01:21.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:01:21.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:01:21.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2024-06-03T05:01:21.548 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:01:21.548 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:01:21.548 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:01:21.548 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:01:21.548 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:01:21.548 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64743 2024-06-03T05:01:21.548 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.64743/$cluster-$name.asok' 2024-06-03T05:01:21.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.64743/$cluster-$name.asok' 2024-06-03T05:01:21.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:01:21.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:01:21.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:01:21.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2024-06-03T05:01:21.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2024-06-03T05:01:21.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:01:21.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:01:21.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:01:21.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:01:21.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:01:21.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:01:21.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/rados-striper/2 2024-06-03T05:01:21.550 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:01:21.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0c3d0dcd-dac2-4b69-9ebf-158f38420381 2024-06-03T05:01:21.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 0c3d0dcd-dac2-4b69-9ebf-158f38420381' 2024-06-03T05:01:21.551 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 0c3d0dcd-dac2-4b69-9ebf-158f38420381 2024-06-03T05:01:21.552 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:01:21.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAhTl1m8ZzrIRAAhsExLzJ6HjzdJpn/kXRprw== 2024-06-03T05:01:21.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAhTl1m8ZzrIRAAhsExLzJ6HjzdJpn/kXRprw=="}' 2024-06-03T05:01:21.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0c3d0dcd-dac2-4b69-9ebf-158f38420381 -i td/rados-striper/2/new.json 2024-06-03T05:01:22.003 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:01:22.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/rados-striper/2/new.json 2024-06-03T05:01:22.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=8960adb3-19d6-4e8f-aca7-e5a821f5594c --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/2 --osd-journal=td/rados-striper/2/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.64743/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAhTl1m8ZzrIRAAhsExLzJ6HjzdJpn/kXRprw== --osd-uuid 0c3d0dcd-dac2-4b69-9ebf-158f38420381 2024-06-03T05:01:22.056 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:01:22.051+0000 7f7e270767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:01:22.089 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:01:22.083+0000 7f7e270767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:01:22.090 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:01:22.087+0000 7f7e270767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:01:22.091 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:01:22.087+0000 7f7e270767c0 -1 bluestore(td/rados-striper/2/block) _read_bdev_label failed to open td/rados-striper/2/block: (2) No such file or directory 2024-06-03T05:01:22.091 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:01:22.087+0000 7f7e270767c0 -1 bluestore(td/rados-striper/2/block) _read_bdev_label failed to open td/rados-striper/2/block: (2) No such file or directory 2024-06-03T05:01:22.091 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:01:22.087+0000 7f7e270767c0 -1 bluestore(td/rados-striper/2/block) _read_bdev_label failed to open td/rados-striper/2/block: (2) No such file or directory 2024-06-03T05:01:22.129 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:01:22.127+0000 7f7e270767c0 -1 bluestore(td/rados-striper/2) _read_fsid unparsable uuid 2024-06-03T05:01:28.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/rados-striper/2/keyring 2024-06-03T05:01:28.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:01:28.596 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-06-03T05:01:28.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-03T05:01:28.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/rados-striper/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:01:28.840 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-06-03T05:01:28.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-03T05:01:28.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=8960adb3-19d6-4e8f-aca7-e5a821f5594c --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/2 --osd-journal=td/rados-striper/2/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.64743/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:01:28.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:01:28.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:01:28.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:01:28.870 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:01:28.867+0000 7fb662b017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:01:28.874 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:01:28.871+0000 7fb662b017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:01:28.875 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:01:28.871+0000 7fb662b017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:01:29.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-03T05:01:29.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:01:29.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T05:01:29.063 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:01:29.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:01:29.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:01:29.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:01:29.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:01:29.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:01:29.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:01:29.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:01:29.468 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:01:29.463+0000 7fb662b017c0 -1 Falling back to public interface 2024-06-03T05:01:30.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:01:30.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:01:30.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:01:30.302 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:01:30.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:01:30.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:01:30.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:01:31.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:01:31.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:01:31.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:01:31.534 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:01:31.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:01:31.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:01:31.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:01:32.228 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:01:32.223+0000 7fb662b017c0 -1 osd.2 0 log_to_monitors true 2024-06-03T05:01:32.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:01:32.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:01:32.755 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:01:32.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:01:32.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:01:32.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:01:33.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:01:34.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:01:34.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:01:34.190 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:01:34.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:01:34.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:01:34.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:01:34.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:01:35.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:01:35.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:01:35.422 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:01:35.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:01:35.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:01:35.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:01:35.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:01:36.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:01:36.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:01:36.656 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:01:36.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:01:36.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:01:36.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:01:36.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:01:37.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:01:37.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:01:37.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:01:37.886 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:01:37.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:01:37.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:01:38.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:01:39.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:01:39.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:01:39.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:01:39.120 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:01:39.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:01:39.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:01:39.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:01:40.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:01:40.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:01:40.348 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:01:40.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:01:40.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:01:40.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:01:40.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:01:41.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:01:41.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:01:41.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:01:41.582 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:01:41.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:01:41.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:01:41.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:01:42.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:01:42.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:01:42.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:01:42.819 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:01:42.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:01:42.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:01:43.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:01:44.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:01:44.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:01:44.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:01:44.054 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:01:44.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:01:44.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:01:44.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:01:45.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:01:45.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:01:45.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:01:45.283 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:01:45.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:01:45.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:01:45.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:01:46.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:01:46.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:01:46.514 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:01:46.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:01:46.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:01:46.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:01:46.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:01:47.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:01:47.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:01:47.747 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:01:47.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:01:47.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:01:47.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:01:47.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:01:48.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:01:48.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:01:48.982 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:01:48.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:01:48.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:01:48.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:01:49.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:01:50.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:01:50.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:01:50.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:01:50.219 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:01:50.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:01:50.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:01:50.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:01:51.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:01:51.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:01:51.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T05:01:51.449 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-06-03T05:01:51.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:01:51.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:01:51.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:01:52.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:01:52.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:01:52.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-03T05:01:52.684 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-06-03T05:01:52.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:01:52.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:01:52.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:01:53.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:01:53.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:01:53.909 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-06-03T05:01:53.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-03T05:01:53.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:01:53.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:01:54.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:01:55.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:01:55.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:01:55.144 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-06-03T05:01:55.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-03T05:01:55.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:01:55.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:01:55.438 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/3463452610,v1:127.0.0.1:6817/3463452610] [v2:127.0.0.1:6818/3463452610,v1:127.0.0.1:6819/3463452610] exists,up 0c3d0dcd-dac2-4b69-9ebf-158f38420381 2024-06-03T05:01:55.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:01:55.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:01:55.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:01:55.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:36: run: create_rbd_pool 2024-06-03T05:01:55.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-03T05:01:55.689 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' does not exist 2024-06-03T05:01:55.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-03T05:01:55.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-03T05:01:56.047 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' already exists 2024-06-03T05:01:56.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T05:01:57.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-03T05:01:59.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:39: run: dd if=/dev/urandom of=td/rados-striper/toyfile bs=1234 count=1 2024-06-03T05:01:59.158 INFO:tasks.workunit.client.0.smithi088.stderr:1+0 records in 2024-06-03T05:01:59.158 INFO:tasks.workunit.client.0.smithi088.stderr:1+0 records out 2024-06-03T05:01:59.158 INFO:tasks.workunit.client.0.smithi088.stderr:1234 bytes (1.2 kB, 1.2 KiB) copied, 0.000132202 s, 9.3 MB/s 2024-06-03T05:01:59.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:42: run: rados --pool rbd --striper put toyfile td/rados-striper/toyfile 2024-06-03T05:02:00.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:45: run: rados --pool rbd --striper stat toyfile 2024-06-03T05:02:00.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:45: run: cut -d , -f 2 2024-06-03T05:02:00.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:46: run: rados --pool rbd stat toyfile.0000000000000000 2024-06-03T05:02:00.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:46: run: cut -d , -f 2 2024-06-03T05:02:00.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:47: run: echo ' size 1234' 2024-06-03T05:02:00.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:48: run: diff -w td/rados-striper/stripedStat td/rados-striper/refstat 2024-06-03T05:02:00.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:49: run: diff -w td/rados-striper/stat td/rados-striper/refstat 2024-06-03T05:02:00.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:50: run: rados --pool rbd stat toyfile 2024-06-03T05:02:00.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:51: run: grep -q 'No such file or directory' td/rados-striper/staterror 2024-06-03T05:02:00.127 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:54: run: rados --pool rbd --striper get toyfile td/rados-striper/stripedGroup 2024-06-03T05:02:00.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:55: run: diff -w td/rados-striper/toyfile td/rados-striper/stripedGroup 2024-06-03T05:02:00.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:56: run: rados --pool rbd get toyfile.0000000000000000 td/rados-striper/nonSTripedGroup 2024-06-03T05:02:00.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:57: run: diff -w td/rados-striper/toyfile td/rados-striper/nonSTripedGroup 2024-06-03T05:02:00.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:60: run: rados --pool rbd --striper truncate toyfile 12 2024-06-03T05:02:00.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:61: run: rados --pool rbd --striper stat toyfile 2024-06-03T05:02:00.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:61: run: cut -d , -f 2 2024-06-03T05:02:00.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:62: run: rados --pool rbd stat toyfile.0000000000000000 2024-06-03T05:02:00.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:62: run: cut -d , -f 2 2024-06-03T05:02:00.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:63: run: echo ' size 12' 2024-06-03T05:02:00.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:64: run: diff -w td/rados-striper/stripedStat td/rados-striper/reftrunc 2024-06-03T05:02:00.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:65: run: diff -w td/rados-striper/stat td/rados-striper/reftrunc 2024-06-03T05:02:00.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:69: run: rados --pool rbd --striper setxattr toyfile somexattr somevalue 2024-06-03T05:02:01.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:70: run: rados --pool rbd --striper getxattr toyfile somexattr 2024-06-03T05:02:01.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:71: run: rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2024-06-03T05:02:01.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:72: run: echo somevalue 2024-06-03T05:02:01.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:73: run: diff -w td/rados-striper/xattrvalue td/rados-striper/refvalue 2024-06-03T05:02:01.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:74: run: diff -w td/rados-striper/xattrvalue2 td/rados-striper/refvalue 2024-06-03T05:02:01.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:75: run: rados --pool rbd --striper listxattr toyfile 2024-06-03T05:02:01.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:76: run: echo somexattr 2024-06-03T05:02:01.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:77: run: diff -w td/rados-striper/xattrlist td/rados-striper/reflist 2024-06-03T05:02:01.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:78: run: rados --pool rbd listxattr toyfile.0000000000000000 2024-06-03T05:02:01.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:78: run: grep -v striper 2024-06-03T05:02:01.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:79: run: diff -w td/rados-striper/xattrlist2 td/rados-striper/reflist 2024-06-03T05:02:01.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:80: run: rados --pool rbd --striper rmxattr toyfile somexattr 2024-06-03T05:02:01.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:82: run: local 'attr_not_found_str=No data available' 2024-06-03T05:02:01.279 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:83: run: uname 2024-06-03T05:02:01.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:83: run: '[' Linux = FreeBSD ']' 2024-06-03T05:02:01.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:85: run: expect_failure td/rados-striper 'No data available' rados --pool rbd --striper getxattr toyfile somexattr 2024-06-03T05:02:01.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1962: expect_failure: local dir=td/rados-striper 2024-06-03T05:02:01.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1963: expect_failure: shift 2024-06-03T05:02:01.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1964: expect_failure: local 'expected=No data available' 2024-06-03T05:02:01.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1965: expect_failure: shift 2024-06-03T05:02:01.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1966: expect_failure: local success 2024-06-03T05:02:01.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: expect_failure: rados --pool rbd --striper getxattr toyfile somexattr 2024-06-03T05:02:01.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: success=false 2024-06-03T05:02:01.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: false 2024-06-03T05:02:01.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: grep --quiet 'No data available' td/rados-striper/out 2024-06-03T05:02:01.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: return 0 2024-06-03T05:02:01.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:87: run: expect_failure td/rados-striper 'No data available' rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2024-06-03T05:02:01.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1962: expect_failure: local dir=td/rados-striper 2024-06-03T05:02:01.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1963: expect_failure: shift 2024-06-03T05:02:01.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1964: expect_failure: local 'expected=No data available' 2024-06-03T05:02:01.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1965: expect_failure: shift 2024-06-03T05:02:01.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1966: expect_failure: local success 2024-06-03T05:02:01.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: expect_failure: rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2024-06-03T05:02:01.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: success=false 2024-06-03T05:02:01.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: false 2024-06-03T05:02:01.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: grep --quiet 'No data available' td/rados-striper/out 2024-06-03T05:02:01.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: return 0 2024-06-03T05:02:01.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:91: run: rados --pool rbd --striper rm toyfile 2024-06-03T05:02:01.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:92: run: expect_failure td/rados-striper 'No such file or directory' rados --pool rbd --striper stat toyfile 2024-06-03T05:02:01.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1962: expect_failure: local dir=td/rados-striper 2024-06-03T05:02:01.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1963: expect_failure: shift 2024-06-03T05:02:01.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1964: expect_failure: local 'expected=No such file or directory' 2024-06-03T05:02:01.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1965: expect_failure: shift 2024-06-03T05:02:01.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1966: expect_failure: local success 2024-06-03T05:02:01.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: expect_failure: rados --pool rbd --striper stat toyfile 2024-06-03T05:02:01.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: success=false 2024-06-03T05:02:01.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: false 2024-06-03T05:02:01.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: grep --quiet 'No such file or directory' td/rados-striper/out 2024-06-03T05:02:01.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: return 0 2024-06-03T05:02:01.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:94: run: expect_failure td/rados-striper 'No such file or directory' rados --pool rbd stat toyfile.0000000000000000 2024-06-03T05:02:01.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1962: expect_failure: local dir=td/rados-striper 2024-06-03T05:02:01.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1963: expect_failure: shift 2024-06-03T05:02:01.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1964: expect_failure: local 'expected=No such file or directory' 2024-06-03T05:02:01.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1965: expect_failure: shift 2024-06-03T05:02:01.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1966: expect_failure: local success 2024-06-03T05:02:01.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: expect_failure: rados --pool rbd stat toyfile.0000000000000000 2024-06-03T05:02:01.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: success=false 2024-06-03T05:02:01.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: false 2024-06-03T05:02:01.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: grep --quiet 'No such file or directory' td/rados-striper/out 2024-06-03T05:02:01.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: return 0 2024-06-03T05:02:01.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:98: run: teardown td/rados-striper 2024-06-03T05:02:01.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/rados-striper 2024-06-03T05:02:01.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:02:01.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/rados-striper KILL 2024-06-03T05:02:01.664 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:02:01.664 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:02:01.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:02:01.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:02:01.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:02:01.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:02:01.772 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:02:01.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:02:01.773 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:02:01.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:02:01.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:02:01.774 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:02:01.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:02:01.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:02:01.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:02:01.776 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:02:01.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:02:01.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:02:01.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/rados-striper 2024-06-03T05:02:01.792 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:02:01.792 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:02:01.792 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64743 2024-06-03T05:02:01.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.64743 2024-06-03T05:02:01.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:02:01.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:02:01.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-06-03T05:02:01.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/rados-striper 0 2024-06-03T05:02:01.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/rados-striper 2024-06-03T05:02:01.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-06-03T05:02:01.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/rados-striper KILL 2024-06-03T05:02:01.794 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:02:01.794 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:02:01.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:02:01.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:02:01.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:02:01.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:02:01.796 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:02:01.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:02:01.797 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:02:01.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:02:01.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:02:01.799 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:02:01.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:02:01.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:02:01.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:02:01.800 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:02:01.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:02:01.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-06-03T05:02:01.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/rados-striper 2024-06-03T05:02:01.804 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:02:01.804 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:02:01.804 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.64743 2024-06-03T05:02:01.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.64743 2024-06-03T05:02:01.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:02:01.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:02:01.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-06-03T05:02:01.806 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-06-03T05:02:01.807 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-06-03T05:02:01.860 INFO:tasks.workunit:Running workunit misc/test-ceph-helpers.sh... 2024-06-03T05:02:01.860 DEBUG:teuthology.orchestra.run.smithi088:workunit test misc/test-ceph-helpers.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dfcc4b3c2c55532c4c04fa47551c9df1dffdf746 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh 2024-06-03T05:02:01.919 INFO:tasks.workunit.client.0.smithi088.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-06-03T05:02:01.923 INFO:tasks.workunit.client.0.smithi088.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-06-03T05:02:01.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: run_tests: export .:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-03T05:02:01.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2328: export: `.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin': not a valid identifier 2024-06-03T05:02:01.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: run_tests: export CEPH_MON=127.0.0.1:7109 2024-06-03T05:02:01.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: run_tests: CEPH_MON=127.0.0.1:7109 2024-06-03T05:02:01.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2331: run_tests: export CEPH_ARGS 2024-06-03T05:02:01.924 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2332: run_tests: uuidgen 2024-06-03T05:02:01.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2332: run_tests: CEPH_ARGS+=' --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none ' 2024-06-03T05:02:01.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2333: run_tests: CEPH_ARGS+='--mon-host=127.0.0.1:7109 ' 2024-06-03T05:02:01.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: run_tests: export CEPH_CONF=/dev/null 2024-06-03T05:02:01.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: run_tests: CEPH_CONF=/dev/null 2024-06-03T05:02:01.925 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2336: run_tests: set 2024-06-03T05:02:01.925 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2336: run_tests: sed -n -e 's/^\(test_[0-9a-z_]*\) .*/\1/p' 2024-06-03T05:02:01.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2336: run_tests: local 'funcs=test_activate_osd 2024-06-03T05:02:01.928 INFO:tasks.workunit.client.0.smithi088.stderr:test_activate_osd_after_mark_down 2024-06-03T05:02:01.928 INFO:tasks.workunit.client.0.smithi088.stderr:test_activate_osd_skip_benchmark 2024-06-03T05:02:01.928 INFO:tasks.workunit.client.0.smithi088.stderr:test_destroy_osd 2024-06-03T05:02:01.928 INFO:tasks.workunit.client.0.smithi088.stderr:test_display_logs 2024-06-03T05:02:01.928 INFO:tasks.workunit.client.0.smithi088.stderr:test_erasure_code_plugin_exists 2024-06-03T05:02:01.929 INFO:tasks.workunit.client.0.smithi088.stderr:test_expect_failure 2024-06-03T05:02:01.929 INFO:tasks.workunit.client.0.smithi088.stderr:test_flush_pg_stats 2024-06-03T05:02:01.929 INFO:tasks.workunit.client.0.smithi088.stderr:test_get_config 2024-06-03T05:02:01.929 INFO:tasks.workunit.client.0.smithi088.stderr:test_get_is_making_recovery_progress 2024-06-03T05:02:01.929 INFO:tasks.workunit.client.0.smithi088.stderr:test_get_last_scrub_stamp 2024-06-03T05:02:01.929 INFO:tasks.workunit.client.0.smithi088.stderr:test_get_not_primary 2024-06-03T05:02:01.929 INFO:tasks.workunit.client.0.smithi088.stderr:test_get_num_active_clean 2024-06-03T05:02:01.929 INFO:tasks.workunit.client.0.smithi088.stderr:test_get_num_active_or_peered 2024-06-03T05:02:01.929 INFO:tasks.workunit.client.0.smithi088.stderr:test_get_num_pgs 2024-06-03T05:02:01.929 INFO:tasks.workunit.client.0.smithi088.stderr:test_get_op_scheduler 2024-06-03T05:02:01.929 INFO:tasks.workunit.client.0.smithi088.stderr:test_get_osd_id_used_by_pgs 2024-06-03T05:02:01.929 INFO:tasks.workunit.client.0.smithi088.stderr:test_get_osds 2024-06-03T05:02:01.930 INFO:tasks.workunit.client.0.smithi088.stderr:test_get_pg 2024-06-03T05:02:01.930 INFO:tasks.workunit.client.0.smithi088.stderr:test_get_primary 2024-06-03T05:02:01.930 INFO:tasks.workunit.client.0.smithi088.stderr:test_get_timeout_delays 2024-06-03T05:02:01.930 INFO:tasks.workunit.client.0.smithi088.stderr:test_is_clean 2024-06-03T05:02:01.930 INFO:tasks.workunit.client.0.smithi088.stderr:test_kill_daemon 2024-06-03T05:02:01.930 INFO:tasks.workunit.client.0.smithi088.stderr:test_kill_daemons 2024-06-03T05:02:01.930 INFO:tasks.workunit.client.0.smithi088.stderr:test_objectstore_tool 2024-06-03T05:02:01.930 INFO:tasks.workunit.client.0.smithi088.stderr:test_pg_schedule_scrub 2024-06-03T05:02:01.930 INFO:tasks.workunit.client.0.smithi088.stderr:test_pg_scrub 2024-06-03T05:02:01.930 INFO:tasks.workunit.client.0.smithi088.stderr:test_repair 2024-06-03T05:02:01.930 INFO:tasks.workunit.client.0.smithi088.stderr:test_run_in_background 2024-06-03T05:02:01.931 INFO:tasks.workunit.client.0.smithi088.stderr:test_run_mon 2024-06-03T05:02:01.931 INFO:tasks.workunit.client.0.smithi088.stderr:test_run_osd 2024-06-03T05:02:01.931 INFO:tasks.workunit.client.0.smithi088.stderr:test_set_config 2024-06-03T05:02:01.931 INFO:tasks.workunit.client.0.smithi088.stderr:test_setup 2024-06-03T05:02:01.931 INFO:tasks.workunit.client.0.smithi088.stderr:test_teardown 2024-06-03T05:02:01.931 INFO:tasks.workunit.client.0.smithi088.stderr:test_wait_background 2024-06-03T05:02:01.931 INFO:tasks.workunit.client.0.smithi088.stderr:test_wait_for_clean 2024-06-03T05:02:01.931 INFO:tasks.workunit.client.0.smithi088.stderr:test_wait_for_health_ok 2024-06-03T05:02:01.931 INFO:tasks.workunit.client.0.smithi088.stderr:test_wait_for_osd 2024-06-03T05:02:01.931 INFO:tasks.workunit.client.0.smithi088.stderr:test_wait_for_peered 2024-06-03T05:02:01.931 INFO:tasks.workunit.client.0.smithi088.stderr:test_wait_for_scrub 2024-06-03T05:02:01.932 INFO:tasks.workunit.client.0.smithi088.stderr:test_wait_osd_id_used_by_pgs' 2024-06-03T05:02:01.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2337: run_tests: local dir=td/ceph-helpers 2024-06-03T05:02:01.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:02:01.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_activate_osd td/ceph-helpers 2024-06-03T05:02:01.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:886: test_activate_osd: local dir=td/ceph-helpers 2024-06-03T05:02:01.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:888: test_activate_osd: setup td/ceph-helpers 2024-06-03T05:02:01.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-03T05:02:01.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-03T05:02:01.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:02:01.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:02:01.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:02:01.933 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:02:01.933 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:02:01.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:02:01.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:02:01.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:02:01.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:02:01.934 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:02:01.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:02:01.935 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:02:01.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:02:01.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:02:01.935 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:02:01.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:02:01.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:02:01.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:02:01.935 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:02:01.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:02:01.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:02:01.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:02:01.937 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:02:01.937 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:02:01.937 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:02:01.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:02:01.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:02:01.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:02:01.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-03T05:02:01.939 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:02:01.940 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:02:01.940 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:02:01.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70897 2024-06-03T05:02:01.941 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:02:01.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-06-03T05:02:01.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-06-03T05:02:01.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:02:01.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-03T05:02:01.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:890: test_activate_osd: run_mon td/ceph-helpers a 2024-06-03T05:02:01.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-03T05:02:01.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:02:01.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:02:01.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:02:01.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-03T05:02:01.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-06-03T05:02:02.345 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:02:02.345 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:02:02.345 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:02:02.345 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:02:02.345 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:02:02.345 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:02:02.345 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:02:02.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-03T05:02:02.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:02:02.555 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:02:02.555 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:02:02.555 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:02:02.555 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:02:02.556 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:02:02.556 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:02:02.556 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:02:02.556 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:02:02.557 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:02:02.557 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:02:02.557 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:02:02.557 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:02:02.557 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:02:02.558 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get fsid 2024-06-03T05:02:02.676 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:02:02.676 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:02:02.676 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:02:02.676 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:02:02.677 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:02:02.677 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:02:02.677 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:02:02.677 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:02:02.677 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:02:02.677 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:02:02.678 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:02:02.678 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:02:02.678 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:02:02.678 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get mon_host 2024-06-03T05:02:02.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:891: test_activate_osd: run_mgr td/ceph-helpers x 2024-06-03T05:02:02.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-03T05:02:02.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T05:02:02.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T05:02:02.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T05:02:02.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-03T05:02:02.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T05:02:03.695 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T05:02:03.695 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:02:03.695 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:02:03.696 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:02:03.696 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:02:03.696 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:02:03.696 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:02:03.696 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:02:03.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:02:03.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:893: test_activate_osd: run_osd td/ceph-helpers 0 2024-06-03T05:02:03.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:02:03.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:02:03.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T05:02:03.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:02:03.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-03T05:02:03.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:02:03.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:02:03.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:02:03.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:02:03.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-03T05:02:03.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-03T05:02:03.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:02:03.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:02:03.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:02:03.735 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:02:03.736 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:02:03.736 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:02:03.736 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:02:03.736 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:02:03.736 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:02:03.736 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:02:03.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:02:03.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:02:03.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:02:03.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:02:03.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:02:03.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:02:03.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:02:03.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:02:03.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:02:03.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:02:03.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:02:03.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:02:03.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-03T05:02:03.737 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:02:03.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8c1dfee9-cf2c-4823-b0da-31d4187d420b 2024-06-03T05:02:03.738 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 8c1dfee9-cf2c-4823-b0da-31d4187d420b 2024-06-03T05:02:03.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8c1dfee9-cf2c-4823-b0da-31d4187d420b' 2024-06-03T05:02:03.738 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:02:03.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBLTl1m1RXPLBAAom9TgA0PW6b+9aVmD3GcFQ== 2024-06-03T05:02:03.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBLTl1m1RXPLBAAom9TgA0PW6b+9aVmD3GcFQ=="}' 2024-06-03T05:02:03.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8c1dfee9-cf2c-4823-b0da-31d4187d420b -i td/ceph-helpers/0/new.json 2024-06-03T05:02:04.069 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:02:04.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-03T05:02:04.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBLTl1m1RXPLBAAom9TgA0PW6b+9aVmD3GcFQ== --osd-uuid 8c1dfee9-cf2c-4823-b0da-31d4187d420b 2024-06-03T05:02:04.124 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:02:04.119+0000 7fe69d1587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:02:04.155 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:02:04.151+0000 7fe69d1587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:02:04.156 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:02:04.151+0000 7fe69d1587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:02:04.157 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:02:04.155+0000 7fe69d1587c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:02:04.157 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:02:04.155+0000 7fe69d1587c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:02:04.157 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:02:04.155+0000 7fe69d1587c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:02:04.194 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:02:04.191+0000 7fe69d1587c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-03T05:02:10.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-03T05:02:10.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:02:10.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T05:02:10.521 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T05:02:10.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:02:11.090 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:02:11.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T05:02:11.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:02:11.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:02:11.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:02:11.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:02:11.123 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:02:11.119+0000 7ff46fcfe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:02:11.155 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:02:11.151+0000 7ff46fcfe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:02:11.156 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:02:11.151+0000 7ff46fcfe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:02:11.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T05:02:11.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:02:11.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:02:11.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:02:11.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:02:11.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:02:11.555 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:02:11.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:02:11.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:02:11.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:02:11.747 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:02:11.743+0000 7ff46fcfe7c0 -1 Falling back to public interface 2024-06-03T05:02:12.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:02:13.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:02:13.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:02:13.029 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:02:13.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:02:13.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:02:13.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:02:13.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:02:14.354 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:02:14.351+0000 7ff46fcfe7c0 -1 osd.0 0 log_to_monitors true 2024-06-03T05:02:14.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:02:14.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:02:14.449 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:02:14.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:02:14.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:02:14.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:02:15.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:02:16.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:02:16.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:02:16.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:02:16.071 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:02:16.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:02:16.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:02:16.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:02:17.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:02:17.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:02:17.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:02:17.547 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:02:17.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:02:17.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:02:17.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:02:18.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:02:18.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:02:18.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:02:18.976 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:02:18.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:02:18.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:02:19.513 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:02:20.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:02:20.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:02:20.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:02:20.515 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:02:20.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:02:20.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:02:20.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:02:21.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:02:21.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:02:21.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:02:21.925 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:02:21.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:02:21.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:02:22.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:02:23.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:02:23.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:02:23.350 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:02:23.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:02:23.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:02:23.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:02:23.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:02:24.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:02:24.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:02:24.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:02:24.777 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:02:24.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:02:24.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:02:25.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:02:26.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:02:26.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:02:26.354 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:02:26.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:02:26.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:02:26.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:02:26.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:02:27.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:02:27.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:02:27.809 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:02:27.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:02:27.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:02:27.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:02:29.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:02:30.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:02:30.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:02:30.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:02:30.187 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:02:30.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:02:30.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:02:30.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:02:31.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:02:31.722 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:02:31.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:02:31.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:02:31.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:02:31.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:02:32.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:02:33.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:02:33.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:02:33.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:02:33.551 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:02:33.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:02:33.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:02:34.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:02:35.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:02:35.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:02:35.303 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:02:35.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:02:35.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:02:35.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:02:36.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:02:37.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:02:37.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:02:37.484 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:02:37.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:02:37.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:02:37.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:02:38.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:02:39.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:02:39.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:02:39.241 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:02:39.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:02:39.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:02:39.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:02:39.651 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/150650150,v1:127.0.0.1:6803/150650150] [v2:127.0.0.1:6804/150650150,v1:127.0.0.1:6805/150650150] exists,up 8c1dfee9-cf2c-4823-b0da-31d4187d420b 2024-06-03T05:02:39.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:02:39.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:02:39.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:02:39.652 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: get_asok_path osd.0 2024-06-03T05:02:39.652 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T05:02:39.652 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T05:02:39.652 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:02:39.652 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:02:39.652 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:02:39.653 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-osd.0.asok 2024-06-03T05:02:39.653 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: CEPH_ARGS= 2024-06-03T05:02:39.653 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: ceph --format=json daemon /tmp/ceph-asok.70897/ceph-osd.0.asok config get osd_max_backfills 2024-06-03T05:02:39.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: local 'backfills={"osd_max_backfills":"1"}' 2024-06-03T05:02:39.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:896: test_activate_osd: echo '{"osd_max_backfills":"1"}' 2024-06-03T05:02:39.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:896: test_activate_osd: grep --quiet osd_max_backfills 2024-06-03T05:02:39.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:898: test_activate_osd: kill_daemons td/ceph-helpers TERM osd 2024-06-03T05:02:39.759 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:02:39.759 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:02:39.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:02:39.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:02:39.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:02:40.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:02:40.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:900: test_activate_osd: activate_osd td/ceph-helpers 0 --osd-max-backfills 20 2024-06-03T05:02:40.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-06-03T05:02:40.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T05:02:40.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T05:02:40.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T05:02:40.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-06-03T05:02:40.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:02:40.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:02:40.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:02:40.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:02:40.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-03T05:02:40.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-03T05:02:40.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T05:02:40.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T05:02:40.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:02:40.070 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T05:02:40.071 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:02:40.071 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:02:40.072 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:02:40.072 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:02:40.072 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:02:40.072 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:02:40.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:02:40.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:02:40.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:02:40.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:02:40.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:02:40.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:02:40.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:02:40.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:02:40.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T05:02:40.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-max-backfills 20' 2024-06-03T05:02:40.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-06-03T05:02:40.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T05:02:40.074 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:02:40.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2024-06-03T05:02:40.074 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-06-03T05:02:40.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T05:02:40.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T05:02:40.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T05:02:40.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T05:02:40.106 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:02:40.103+0000 7f99db5827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:02:40.109 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:02:40.107+0000 7f99db5827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:02:40.110 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:02:40.107+0000 7f99db5827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:02:40.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T05:02:40.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:02:40.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:02:40.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:02:40.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:02:40.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:02:40.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:02:40.530 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:02:40.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:02:40.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:02:40.642 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:02:40.639+0000 7f99db5827c0 -1 Falling back to public interface 2024-06-03T05:02:40.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:02:41.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:02:41.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:02:41.960 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:02:41.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:02:41.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:02:41.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:02:42.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:02:43.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:02:43.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:02:43.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:02:43.410 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:02:43.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:02:43.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:02:43.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:02:43.895 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:02:43.891+0000 7f99db5827c0 -1 osd.0 6 log_to_monitors true 2024-06-03T05:02:44.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:02:44.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:02:44.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:02:44.837 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:02:44.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:02:44.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:02:45.243 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6802/115163483,v1:127.0.0.1:6803/115163483] [v2:127.0.0.1:6804/115163483,v1:127.0.0.1:6805/115163483] exists,up 8c1dfee9-cf2c-4823-b0da-31d4187d420b 2024-06-03T05:02:45.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:02:45.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:02:45.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:02:45.245 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: get_asok_path osd.0 2024-06-03T05:02:45.245 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T05:02:45.245 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T05:02:45.245 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:02:45.245 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:02:45.246 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:02:45.246 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-osd.0.asok 2024-06-03T05:02:45.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: CEPH_ARGS= 2024-06-03T05:02:45.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: ceph --format=json daemon /tmp/ceph-asok.70897/ceph-osd.0.asok config get osd_max_backfills 2024-06-03T05:02:45.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: local 'backfills={"osd_max_backfills":"20"}' 2024-06-03T05:02:45.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:903: test_activate_osd: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2024-06-03T05:02:45.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:905: test_activate_osd: teardown td/ceph-helpers 2024-06-03T05:02:45.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:02:45.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:02:45.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:02:45.366 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:02:45.366 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:02:45.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:02:45.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:02:45.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:02:45.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:02:45.474 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:02:45.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:02:45.475 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:02:45.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:02:45.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:02:45.476 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:02:45.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:02:45.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:02:45.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:02:45.477 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:02:45.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:02:45.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:02:45.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:02:45.485 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:02:45.485 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:02:45.485 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:02:45.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:02:45.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:02:45.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:02:45.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:02:45.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_activate_osd_after_mark_down td/ceph-helpers 2024-06-03T05:02:45.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:909: test_activate_osd_after_mark_down: local dir=td/ceph-helpers 2024-06-03T05:02:45.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:911: test_activate_osd_after_mark_down: setup td/ceph-helpers 2024-06-03T05:02:45.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-03T05:02:45.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-03T05:02:45.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:02:45.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:02:45.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:02:45.488 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:02:45.488 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:02:45.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:02:45.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:02:45.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:02:45.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:02:45.488 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:02:45.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:02:45.489 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:02:45.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:02:45.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:02:45.491 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:02:45.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:02:45.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:02:45.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:02:45.492 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:02:45.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:02:45.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:02:45.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:02:45.494 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:02:45.495 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:02:45.495 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:02:45.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:02:45.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:02:45.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:02:45.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-03T05:02:45.496 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:02:45.497 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:02:45.497 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:02:45.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70897 2024-06-03T05:02:45.498 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:02:45.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:02:45.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:02:45.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-03T05:02:45.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:913: test_activate_osd_after_mark_down: run_mon td/ceph-helpers a 2024-06-03T05:02:45.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-03T05:02:45.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:02:45.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:02:45.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:02:45.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-03T05:02:45.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-06-03T05:02:45.894 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:02:45.894 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:02:45.894 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:02:45.894 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:02:45.894 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:02:45.895 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:02:45.895 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:02:45.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-03T05:02:46.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:02:46.138 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:02:46.138 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:02:46.138 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:02:46.138 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:02:46.139 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:02:46.140 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:02:46.140 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:02:46.140 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:02:46.140 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:02:46.140 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:02:46.140 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:02:46.140 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:02:46.140 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:02:46.140 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get fsid 2024-06-03T05:02:46.250 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:02:46.250 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:02:46.250 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:02:46.250 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:02:46.251 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:02:46.251 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:02:46.251 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:02:46.252 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:02:46.252 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:02:46.252 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:02:46.252 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:02:46.252 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:02:46.252 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:02:46.252 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get mon_host 2024-06-03T05:02:46.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: test_activate_osd_after_mark_down: run_mgr td/ceph-helpers x 2024-06-03T05:02:46.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-03T05:02:46.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T05:02:46.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T05:02:46.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T05:02:46.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-03T05:02:46.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T05:02:47.285 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T05:02:47.286 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:02:47.286 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:02:47.286 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:02:47.286 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:02:47.287 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:02:47.287 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:02:47.287 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:02:47.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:02:47.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: test_activate_osd_after_mark_down: run_osd td/ceph-helpers 0 2024-06-03T05:02:47.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:02:47.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:02:47.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T05:02:47.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:02:47.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-03T05:02:47.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:02:47.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:02:47.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:02:47.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:02:47.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-03T05:02:47.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-03T05:02:47.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:02:47.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:02:47.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:02:47.322 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:02:47.322 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:02:47.322 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:02:47.322 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:02:47.322 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:02:47.323 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:02:47.323 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:02:47.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:02:47.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:02:47.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:02:47.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:02:47.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:02:47.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:02:47.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:02:47.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:02:47.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:02:47.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:02:47.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:02:47.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:02:47.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-03T05:02:47.324 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:02:47.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fcb905cd-c6f0-4693-b1c8-01c4cd4a5b43 2024-06-03T05:02:47.324 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 fcb905cd-c6f0-4693-b1c8-01c4cd4a5b43 2024-06-03T05:02:47.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 fcb905cd-c6f0-4693-b1c8-01c4cd4a5b43' 2024-06-03T05:02:47.325 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:02:47.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB3Tl1m3b0+FBAAJMlEjzuWR5qfi7r7LXn+Nw== 2024-06-03T05:02:47.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB3Tl1m3b0+FBAAJMlEjzuWR5qfi7r7LXn+Nw=="}' 2024-06-03T05:02:47.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fcb905cd-c6f0-4693-b1c8-01c4cd4a5b43 -i td/ceph-helpers/0/new.json 2024-06-03T05:02:47.659 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:02:47.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-03T05:02:47.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB3Tl1m3b0+FBAAJMlEjzuWR5qfi7r7LXn+Nw== --osd-uuid fcb905cd-c6f0-4693-b1c8-01c4cd4a5b43 2024-06-03T05:02:47.712 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:02:47.707+0000 7fea9c6197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:02:47.745 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:02:47.739+0000 7fea9c6197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:02:47.746 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:02:47.743+0000 7fea9c6197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:02:47.747 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:02:47.743+0000 7fea9c6197c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:02:47.747 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:02:47.743+0000 7fea9c6197c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:02:47.747 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:02:47.743+0000 7fea9c6197c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:02:47.776 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:02:47.771+0000 7fea9c6197c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-03T05:02:54.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-03T05:02:54.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:02:54.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T05:02:54.218 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T05:02:54.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:02:54.768 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:02:54.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T05:02:54.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:02:54.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:02:54.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:02:54.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:02:54.800 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:02:54.795+0000 7f083de4d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:02:54.802 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:02:54.799+0000 7f083de4d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:02:54.803 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:02:54.799+0000 7f083de4d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:02:55.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T05:02:55.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:02:55.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:02:55.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:02:55.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:02:55.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:02:55.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:02:55.218 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:02:55.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:02:55.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:02:55.374 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:02:55.371+0000 7f083de4d7c0 -1 Falling back to public interface 2024-06-03T05:02:55.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:02:56.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:02:56.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:02:56.638 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:02:56.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:02:56.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:02:56.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:02:57.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:02:58.025 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:02:58.023+0000 7f083de4d7c0 -1 osd.0 0 log_to_monitors true 2024-06-03T05:02:58.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:02:58.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:02:58.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:02:58.047 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:02:58.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:02:58.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:02:58.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:02:59.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:02:59.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:02:59.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:02:59.710 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:02:59.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:02:59.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:03:00.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:03:01.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:03:01.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:03:01.158 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:03:01.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:03:01.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:03:01.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:03:01.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:03:02.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:03:02.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:03:02.589 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:03:02.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:03:02.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:03:02.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:03:03.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:03:04.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:03:04.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:03:04.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:03:04.187 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:03:04.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:03:04.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:03:04.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:03:05.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:03:05.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:03:05.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:03:05.649 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:03:05.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:03:05.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:03:06.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:03:07.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:03:07.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:03:07.173 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:03:07.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:03:07.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:03:07.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:03:07.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:03:08.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:03:08.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:03:08.646 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:03:08.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:03:08.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:03:08.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:03:09.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:03:10.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:03:10.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:03:10.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:03:10.194 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:03:10.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:03:10.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:03:10.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:03:11.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:03:11.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:03:11.623 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:03:11.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:03:11.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:03:11.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:03:12.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:03:13.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:03:13.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:03:13.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:03:13.374 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:03:13.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:03:13.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:03:14.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:03:15.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:03:15.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:03:15.135 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:03:15.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:03:15.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:03:15.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:03:16.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:03:17.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:03:17.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:03:17.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:03:17.072 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:03:17.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:03:17.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:03:17.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:03:18.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:03:18.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:03:18.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:03:18.884 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:03:18.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:03:18.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:03:20.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:03:21.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:03:21.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:03:21.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:03:21.048 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:03:21.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:03:21.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:03:22.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:03:23.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:03:23.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:03:23.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:03:23.114 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:03:23.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:03:23.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:03:23.531 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1694056409,v1:127.0.0.1:6803/1694056409] [v2:127.0.0.1:6804/1694056409,v1:127.0.0.1:6805/1694056409] exists,up fcb905cd-c6f0-4693-b1c8-01c4cd4a5b43 2024-06-03T05:03:23.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:03:23.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:03:23.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:03:23.532 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: get_asok_path osd.0 2024-06-03T05:03:23.532 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T05:03:23.532 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T05:03:23.533 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:03:23.533 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:03:23.533 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:03:23.533 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-osd.0.asok 2024-06-03T05:03:23.534 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: CEPH_ARGS= 2024-06-03T05:03:23.534 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: ceph --format=json daemon /tmp/ceph-asok.70897/ceph-osd.0.asok config get osd_max_backfills 2024-06-03T05:03:23.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: local 'backfills={"osd_max_backfills":"1"}' 2024-06-03T05:03:23.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: test_activate_osd_after_mark_down: echo '{"osd_max_backfills":"1"}' 2024-06-03T05:03:23.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: test_activate_osd_after_mark_down: grep --quiet osd_max_backfills 2024-06-03T05:03:23.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: test_activate_osd_after_mark_down: kill_daemons td/ceph-helpers TERM osd 2024-06-03T05:03:23.657 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:03:23.657 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:03:23.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:03:23.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:03:23.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:03:23.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:03:23.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: test_activate_osd_after_mark_down: ceph osd down 0 2024-06-03T05:03:24.416 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already down. 2024-06-03T05:03:24.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: test_activate_osd_after_mark_down: wait_for_osd down 0 2024-06-03T05:03:24.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2024-06-03T05:03:24.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:03:24.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:03:24.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:03:24.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:03:24.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:03:24.433 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:03:24.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:03:24.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2024-06-03T05:03:24.850 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 7 last_clean_interval [0,0) [v2:127.0.0.1:6802/1694056409,v1:127.0.0.1:6803/1694056409] [v2:127.0.0.1:6804/1694056409,v1:127.0.0.1:6805/1694056409] exists fcb905cd-c6f0-4693-b1c8-01c4cd4a5b43 2024-06-03T05:03:24.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:03:24.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:03:24.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:03:24.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: test_activate_osd_after_mark_down: activate_osd td/ceph-helpers 0 --osd-max-backfills 20 2024-06-03T05:03:24.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-06-03T05:03:24.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T05:03:24.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T05:03:24.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T05:03:24.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-06-03T05:03:24.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:03:24.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:03:24.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:03:24.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:03:24.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-03T05:03:24.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-03T05:03:24.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T05:03:24.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T05:03:24.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:03:24.853 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T05:03:24.853 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:03:24.853 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:03:24.855 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:03:24.855 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:03:24.855 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:03:24.855 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:03:24.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:03:24.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:03:24.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:03:24.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:03:24.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:03:24.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:03:24.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:03:24.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:03:24.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T05:03:24.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-max-backfills 20' 2024-06-03T05:03:24.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-06-03T05:03:24.856 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:03:24.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T05:03:24.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2024-06-03T05:03:24.857 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-06-03T05:03:24.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T05:03:24.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T05:03:24.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T05:03:24.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T05:03:24.887 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:03:24.883+0000 7f82af4cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:03:24.895 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:03:24.891+0000 7f82af4cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:03:24.896 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:03:24.891+0000 7f82af4cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:03:25.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T05:03:25.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:03:25.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:03:25.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:03:25.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:03:25.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:03:25.252 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:03:25.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:03:25.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:03:25.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:03:25.447 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:03:25.443+0000 7f82af4cc7c0 -1 Falling back to public interface 2024-06-03T05:03:25.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:03:26.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:03:26.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:03:26.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:03:26.693 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:03:26.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:03:26.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:03:27.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:03:28.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:03:28.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:03:28.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:03:28.107 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:03:28.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:03:28.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:03:28.621 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:03:28.615+0000 7f82af4cc7c0 -1 osd.0 6 log_to_monitors true 2024-06-03T05:03:28.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:03:29.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:03:29.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:03:29.642 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:03:29.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:03:29.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:03:29.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:03:30.051 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6802/1688110042,v1:127.0.0.1:6803/1688110042] [v2:127.0.0.1:6804/1688110042,v1:127.0.0.1:6805/1688110042] exists,up fcb905cd-c6f0-4693-b1c8-01c4cd4a5b43 2024-06-03T05:03:30.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:03:30.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:03:30.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:03:30.052 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: get_asok_path osd.0 2024-06-03T05:03:30.053 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T05:03:30.053 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T05:03:30.053 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:03:30.053 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:03:30.053 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:03:30.053 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-osd.0.asok 2024-06-03T05:03:30.053 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: CEPH_ARGS= 2024-06-03T05:03:30.054 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: ceph --format=json daemon /tmp/ceph-asok.70897/ceph-osd.0.asok config get osd_max_backfills 2024-06-03T05:03:30.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: local 'backfills={"osd_max_backfills":"20"}' 2024-06-03T05:03:30.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: test_activate_osd_after_mark_down: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2024-06-03T05:03:30.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: test_activate_osd_after_mark_down: teardown td/ceph-helpers 2024-06-03T05:03:30.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:03:30.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:03:30.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:03:30.173 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:03:30.173 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:03:30.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:03:30.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:03:30.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:03:30.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:03:30.282 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:03:30.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:03:30.283 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:03:30.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:03:30.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:03:30.284 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:03:30.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:03:30.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:03:30.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:03:30.286 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:03:30.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:03:30.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:03:30.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:03:30.294 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:03:30.294 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:03:30.294 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:03:30.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:03:30.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:03:30.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:03:30.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:03:30.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_activate_osd_skip_benchmark td/ceph-helpers 2024-06-03T05:03:30.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:934: test_activate_osd_skip_benchmark: local dir=td/ceph-helpers 2024-06-03T05:03:30.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:936: test_activate_osd_skip_benchmark: setup td/ceph-helpers 2024-06-03T05:03:30.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-03T05:03:30.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-03T05:03:30.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:03:30.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:03:30.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:03:30.297 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:03:30.297 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:03:30.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:03:30.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:03:30.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:03:30.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:03:30.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:03:30.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:03:30.299 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:03:30.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:03:30.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:03:30.301 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:03:30.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:03:30.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:03:30.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:03:30.302 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:03:30.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:03:30.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:03:30.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:03:30.305 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:03:30.305 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:03:30.305 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:03:30.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:03:30.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:03:30.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:03:30.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-03T05:03:30.307 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:03:30.307 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:03:30.308 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:03:30.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70897 2024-06-03T05:03:30.309 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:03:30.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:03:30.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:03:30.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-03T05:03:30.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:938: test_activate_osd_skip_benchmark: run_mon td/ceph-helpers a 2024-06-03T05:03:30.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-03T05:03:30.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:03:30.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:03:30.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:03:30.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-03T05:03:30.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-06-03T05:03:30.747 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:03:30.747 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:03:30.747 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:03:30.747 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:03:30.747 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:03:30.747 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:03:30.747 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:03:30.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-03T05:03:30.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:03:30.981 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:03:30.981 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:03:30.981 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:03:30.981 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:03:30.982 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:03:30.983 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:03:30.983 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:03:30.983 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:03:30.983 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:03:30.983 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:03:30.983 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:03:30.983 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:03:30.983 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:03:30.983 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get fsid 2024-06-03T05:03:31.095 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:03:31.095 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:03:31.095 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:03:31.095 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:03:31.095 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:03:31.096 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:03:31.096 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:03:31.096 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:03:31.096 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:03:31.096 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:03:31.096 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:03:31.096 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:03:31.097 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:03:31.097 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get mon_host 2024-06-03T05:03:31.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:939: test_activate_osd_skip_benchmark: run_mgr td/ceph-helpers x 2024-06-03T05:03:31.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-03T05:03:31.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T05:03:31.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T05:03:31.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T05:03:31.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-03T05:03:31.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T05:03:32.171 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T05:03:32.171 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:03:32.171 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:03:32.171 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:03:32.171 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:03:32.171 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:03:32.172 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:03:32.172 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:03:32.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:03:32.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:942: test_activate_osd_skip_benchmark: run_osd td/ceph-helpers 0 --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-06-03T05:03:32.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:03:32.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:03:32.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T05:03:32.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:03:32.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-03T05:03:32.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:03:32.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:03:32.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:03:32.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:03:32.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-03T05:03:32.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-03T05:03:32.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:03:32.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:03:32.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:03:32.213 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:03:32.213 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:03:32.214 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:03:32.214 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:03:32.214 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:03:32.214 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:03:32.214 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:03:32.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:03:32.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:03:32.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:03:32.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:03:32.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:03:32.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:03:32.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:03:32.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:03:32.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:03:32.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:03:32.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:03:32.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true' 2024-06-03T05:03:32.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-03T05:03:32.215 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:03:32.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3440087f-a84c-4556-930c-71a1a8f882ef 2024-06-03T05:03:32.215 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 3440087f-a84c-4556-930c-71a1a8f882ef 2024-06-03T05:03:32.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3440087f-a84c-4556-930c-71a1a8f882ef' 2024-06-03T05:03:32.216 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:03:32.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCkTl1m+KKlDRAAn7AqnCUEltERdDEt8iub4g== 2024-06-03T05:03:32.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCkTl1m+KKlDRAAn7AqnCUEltERdDEt8iub4g=="}' 2024-06-03T05:03:32.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3440087f-a84c-4556-930c-71a1a8f882ef -i td/ceph-helpers/0/new.json 2024-06-03T05:03:32.550 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:03:32.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-03T05:03:32.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true --mkfs --key AQCkTl1m+KKlDRAAn7AqnCUEltERdDEt8iub4g== --osd-uuid 3440087f-a84c-4556-930c-71a1a8f882ef 2024-06-03T05:03:32.601 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:03:32.599+0000 7f72ee7877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:03:32.627 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:03:32.623+0000 7f72ee7877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:03:32.628 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:03:32.623+0000 7f72ee7877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:03:32.629 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:03:32.627+0000 7f72ee7877c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:03:32.629 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:03:32.627+0000 7f72ee7877c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:03:32.629 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:03:32.627+0000 7f72ee7877c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:03:32.676 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:03:32.671+0000 7f72ee7877c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-03T05:03:39.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-03T05:03:39.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:03:39.065 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T05:03:39.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T05:03:39.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:03:39.619 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:03:39.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T05:03:39.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-06-03T05:03:39.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:03:39.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:03:39.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:03:39.652 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:03:39.647+0000 7fe11fbf07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:03:39.654 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:03:39.651+0000 7fe11fbf07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:03:39.656 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:03:39.651+0000 7fe11fbf07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:03:40.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T05:03:40.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:03:40.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:03:40.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:03:40.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:03:40.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:03:40.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:03:40.017 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:03:40.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:03:40.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:03:40.232 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:03:40.227+0000 7fe11fbf07c0 -1 Falling back to public interface 2024-06-03T05:03:40.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:03:41.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:03:41.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:03:41.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:03:41.455 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:03:41.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:03:41.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:03:41.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:03:42.839 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:03:42.835+0000 7fe11fbf07c0 -1 osd.0 0 log_to_monitors true 2024-06-03T05:03:42.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:03:42.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:03:42.860 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:03:42.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:03:42.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:03:42.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:03:43.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:03:44.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:03:44.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:03:44.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:03:44.415 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:03:44.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:03:44.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:03:44.818 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4124543889,v1:127.0.0.1:6803/4124543889] [v2:127.0.0.1:6804/4124543889,v1:127.0.0.1:6805/4124543889] exists,up 3440087f-a84c-4556-930c-71a1a8f882ef 2024-06-03T05:03:44.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:03:44.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:03:44.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:03:44.819 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-06-03T05:03:44.819 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T05:03:44.819 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T05:03:44.820 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:03:44.820 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:03:44.820 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:03:44.821 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-osd.0.asok 2024-06-03T05:03:44.821 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-06-03T05:03:44.821 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.70897/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_hdd 2024-06-03T05:03:44.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: local 'max_iops_hdd_def={"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2024-06-03T05:03:44.942 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-06-03T05:03:44.942 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T05:03:44.942 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T05:03:44.943 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:03:44.943 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:03:44.943 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:03:44.943 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-osd.0.asok 2024-06-03T05:03:44.943 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-06-03T05:03:44.944 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.70897/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_ssd 2024-06-03T05:03:45.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: local 'max_iops_ssd_def={"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2024-06-03T05:03:45.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:949: test_activate_osd_skip_benchmark: kill_daemons td/ceph-helpers TERM osd 2024-06-03T05:03:45.062 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:03:45.062 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:03:45.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:03:45.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:03:45.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:03:46.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:03:46.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:950: test_activate_osd_skip_benchmark: ceph osd down 0 2024-06-03T05:03:46.780 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already down. 2024-06-03T05:03:46.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:951: test_activate_osd_skip_benchmark: wait_for_osd down 0 2024-06-03T05:03:46.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2024-06-03T05:03:46.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:03:46.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:03:46.798 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:03:46.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:03:46.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:03:46.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:03:46.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:03:46.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2024-06-03T05:03:47.209 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 7 last_clean_interval [0,0) [v2:127.0.0.1:6802/4124543889,v1:127.0.0.1:6803/4124543889] [v2:127.0.0.1:6804/4124543889,v1:127.0.0.1:6805/4124543889] exists 3440087f-a84c-4556-930c-71a1a8f882ef 2024-06-03T05:03:47.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:03:47.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:03:47.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:03:47.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:955: test_activate_osd_skip_benchmark: activate_osd td/ceph-helpers 0 --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-06-03T05:03:47.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-06-03T05:03:47.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T05:03:47.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T05:03:47.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T05:03:47.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-06-03T05:03:47.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:03:47.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:03:47.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:03:47.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:03:47.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-03T05:03:47.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-03T05:03:47.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T05:03:47.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T05:03:47.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:03:47.215 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T05:03:47.215 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:03:47.215 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:03:47.215 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:03:47.215 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:03:47.215 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:03:47.216 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:03:47.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:03:47.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:03:47.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:03:47.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:03:47.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:03:47.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:03:47.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:03:47.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:03:47.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T05:03:47.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true' 2024-06-03T05:03:47.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-06-03T05:03:47.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T05:03:47.217 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:03:47.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-06-03T05:03:47.217 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-06-03T05:03:47.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T05:03:47.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T05:03:47.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T05:03:47.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T05:03:47.250 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:03:47.247+0000 7f0372ff17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:03:47.252 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:03:47.247+0000 7f0372ff17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:03:47.254 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:03:47.251+0000 7f0372ff17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:03:47.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T05:03:47.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:03:47.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:03:47.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:03:47.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:03:47.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:03:47.626 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:03:47.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:03:47.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:03:47.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:03:47.799 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:03:47.795+0000 7f0372ff17c0 -1 Falling back to public interface 2024-06-03T05:03:48.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:03:49.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:03:49.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:03:49.031 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:03:49.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:03:49.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:03:49.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:03:49.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:03:50.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:03:50.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:03:50.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:03:50.433 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:03:50.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:03:50.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:03:50.811 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:03:50.807+0000 7f0372ff17c0 -1 osd.0 6 log_to_monitors true 2024-06-03T05:03:50.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:03:51.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:03:51.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:03:51.854 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:03:51.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:03:51.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:03:51.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:03:52.261 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6802/3216087117,v1:127.0.0.1:6803/3216087117] [v2:127.0.0.1:6804/3216087117,v1:127.0.0.1:6805/3216087117] exists,up 3440087f-a84c-4556-930c-71a1a8f882ef 2024-06-03T05:03:52.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:03:52.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:03:52.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:03:52.263 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-06-03T05:03:52.263 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T05:03:52.263 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T05:03:52.263 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:03:52.263 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:03:52.263 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:03:52.264 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-osd.0.asok 2024-06-03T05:03:52.264 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-06-03T05:03:52.264 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.70897/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_hdd 2024-06-03T05:03:52.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: local 'max_iops_hdd_after_boot={"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2024-06-03T05:03:52.386 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-06-03T05:03:52.386 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T05:03:52.386 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T05:03:52.386 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:03:52.386 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:03:52.386 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:03:52.386 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-osd.0.asok 2024-06-03T05:03:52.386 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-06-03T05:03:52.387 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.70897/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_ssd 2024-06-03T05:03:52.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: local 'max_iops_ssd_after_boot={"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2024-06-03T05:03:52.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: test_activate_osd_skip_benchmark: test '{"osd_mclock_max_capacity_iops_hdd":"315.000000"}' = '{"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2024-06-03T05:03:52.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: test_activate_osd_skip_benchmark: test '{"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' = '{"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2024-06-03T05:03:52.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: test_activate_osd_skip_benchmark: teardown td/ceph-helpers 2024-06-03T05:03:52.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:03:52.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:03:52.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:03:52.512 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:03:52.513 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:03:52.513 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:03:52.513 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:03:52.513 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:03:52.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:03:52.619 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:03:52.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:03:52.620 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:03:52.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:03:52.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:03:52.622 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:03:52.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:03:52.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:03:52.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:03:52.623 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:03:52.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:03:52.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:03:52.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:03:52.628 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:03:52.628 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:03:52.628 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:03:52.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:03:52.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:03:52.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:03:52.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:03:52.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_destroy_osd td/ceph-helpers 2024-06-03T05:03:52.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:796: test_destroy_osd: local dir=td/ceph-helpers 2024-06-03T05:03:52.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:798: test_destroy_osd: setup td/ceph-helpers 2024-06-03T05:03:52.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-03T05:03:52.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-03T05:03:52.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:03:52.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:03:52.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:03:52.631 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:03:52.631 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:03:52.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:03:52.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:03:52.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:03:52.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:03:52.632 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:03:52.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:03:52.632 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:03:52.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:03:52.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:03:52.633 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:03:52.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:03:52.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:03:52.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:03:52.635 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:03:52.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:03:52.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:03:52.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:03:52.637 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:03:52.637 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:03:52.637 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:03:52.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:03:52.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:03:52.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:03:52.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-03T05:03:52.639 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:03:52.639 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:03:52.639 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:03:52.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70897 2024-06-03T05:03:52.641 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:03:52.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:03:52.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:03:52.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-03T05:03:52.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:799: test_destroy_osd: run_mon td/ceph-helpers a 2024-06-03T05:03:52.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-03T05:03:52.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:03:52.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:03:52.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:03:52.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-03T05:03:52.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-06-03T05:03:53.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:03:53.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:03:53.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:03:53.093 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:03:53.093 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:03:53.093 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:03:53.094 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:03:53.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-03T05:03:53.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:03:53.320 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:03:53.320 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:03:53.320 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:03:53.320 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:03:53.322 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:03:53.322 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:03:53.322 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:03:53.322 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:03:53.322 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:03:53.322 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:03:53.322 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:03:53.322 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:03:53.323 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:03:53.323 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get fsid 2024-06-03T05:03:53.432 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:03:53.432 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:03:53.433 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:03:53.433 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:03:53.434 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:03:53.434 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:03:53.434 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:03:53.434 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:03:53.434 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:03:53.434 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:03:53.434 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:03:53.434 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:03:53.434 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:03:53.435 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get mon_host 2024-06-03T05:03:53.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:800: test_destroy_osd: run_mgr td/ceph-helpers x 2024-06-03T05:03:53.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-03T05:03:53.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T05:03:53.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T05:03:53.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T05:03:53.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-03T05:03:53.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T05:03:54.435 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T05:03:54.435 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:03:54.435 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:03:54.435 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:03:54.435 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:03:54.435 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:03:54.436 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:03:54.436 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:03:54.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:03:54.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:801: test_destroy_osd: run_osd td/ceph-helpers 0 2024-06-03T05:03:54.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:03:54.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:03:54.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T05:03:54.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:03:54.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-03T05:03:54.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:03:54.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:03:54.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:03:54.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:03:54.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-03T05:03:54.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-03T05:03:54.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:03:54.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:03:54.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:03:54.472 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:03:54.473 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:03:54.473 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:03:54.473 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:03:54.473 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:03:54.473 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:03:54.473 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:03:54.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:03:54.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:03:54.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:03:54.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:03:54.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:03:54.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:03:54.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:03:54.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:03:54.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:03:54.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:03:54.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:03:54.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:03:54.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-03T05:03:54.474 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:03:54.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d37e5cb6-4fe4-483e-826c-7510355c92a0 2024-06-03T05:03:54.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d37e5cb6-4fe4-483e-826c-7510355c92a0' 2024-06-03T05:03:54.475 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 d37e5cb6-4fe4-483e-826c-7510355c92a0 2024-06-03T05:03:54.475 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:03:54.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC6Tl1mKOA5HRAAUgFI5JkM75QIEzVCYQnwNg== 2024-06-03T05:03:54.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC6Tl1mKOA5HRAAUgFI5JkM75QIEzVCYQnwNg=="}' 2024-06-03T05:03:54.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d37e5cb6-4fe4-483e-826c-7510355c92a0 -i td/ceph-helpers/0/new.json 2024-06-03T05:03:54.808 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:03:54.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-03T05:03:54.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC6Tl1mKOA5HRAAUgFI5JkM75QIEzVCYQnwNg== --osd-uuid d37e5cb6-4fe4-483e-826c-7510355c92a0 2024-06-03T05:03:54.861 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:03:54.855+0000 7f3332b447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:03:54.877 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:03:54.875+0000 7f3332b447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:03:54.878 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:03:54.875+0000 7f3332b447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:03:54.879 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:03:54.875+0000 7f3332b447c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:03:54.879 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:03:54.875+0000 7f3332b447c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:03:54.879 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:03:54.875+0000 7f3332b447c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:03:54.916 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:03:54.911+0000 7f3332b447c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-03T05:04:01.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-03T05:04:01.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:04:01.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T05:04:01.188 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T05:04:01.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:04:01.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T05:04:01.734 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:04:01.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:04:01.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:04:01.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:04:01.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:04:01.764 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:04:01.759+0000 7f92f22c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:04:01.791 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:04:01.787+0000 7f92f22c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:04:01.793 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:04:01.791+0000 7f92f22c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:04:02.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T05:04:02.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:04:02.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:04:02.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:04:02.157 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:04:02.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:04:02.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:04:02.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:04:02.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:04:02.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:04:02.358 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:04:02.355+0000 7f92f22c77c0 -1 Falling back to public interface 2024-06-03T05:04:02.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:04:03.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:04:03.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:04:03.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:04:03.572 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:04:03.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:04:03.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:04:03.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:04:04.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:04:04.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:04:04.974 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:04:04.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:04:04.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:04:04.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:04:04.982 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:04:04.979+0000 7f92f22c77c0 -1 osd.0 0 log_to_monitors true 2024-06-03T05:04:05.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:04:06.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:04:06.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:04:06.643 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:04:06.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:04:06.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:04:06.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:04:07.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:04:08.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:04:08.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:04:08.090 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:04:08.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:04:08.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:04:08.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:04:08.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:04:09.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:04:09.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:04:09.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:04:09.538 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:04:09.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:04:09.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:04:10.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:04:11.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:04:11.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:04:11.094 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:04:11.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:04:11.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:04:11.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:04:11.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:04:12.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:04:12.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:04:12.532 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:04:12.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:04:12.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:04:12.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:04:13.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:04:14.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:04:14.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:04:14.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:04:14.181 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:04:14.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:04:14.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:04:14.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:04:15.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:04:15.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:04:15.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:04:15.722 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:04:15.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:04:15.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:04:16.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:04:17.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:04:17.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:04:17.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:04:17.141 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:04:17.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:04:17.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:04:17.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:04:18.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:04:18.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:04:18.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:04:18.580 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:04:18.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:04:18.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:04:19.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:04:20.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:04:20.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:04:20.222 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:04:20.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:04:20.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:04:20.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:04:21.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:04:22.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:04:22.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:04:22.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:04:22.153 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:04:22.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:04:22.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:04:22.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:04:23.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:04:23.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:04:23.904 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:04:23.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:04:23.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:04:23.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:04:24.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:04:25.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:04:25.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:04:25.796 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:04:25.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:04:25.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:04:25.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:04:26.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:04:27.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:04:27.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:04:27.922 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:04:27.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:04:27.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:04:27.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:04:28.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:04:29.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:04:29.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:04:29.816 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:04:29.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:04:29.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:04:29.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:04:30.309 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1018062984,v1:127.0.0.1:6803/1018062984] [v2:127.0.0.1:6804/1018062984,v1:127.0.0.1:6805/1018062984] exists,up d37e5cb6-4fe4-483e-826c-7510355c92a0 2024-06-03T05:04:30.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:04:30.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:04:30.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:04:30.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:802: test_destroy_osd: destroy_osd td/ceph-helpers 0 2024-06-03T05:04:30.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:784: destroy_osd: local dir=td/ceph-helpers 2024-06-03T05:04:30.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:785: destroy_osd: local id=0 2024-06-03T05:04:30.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:787: destroy_osd: ceph osd out osd.0 2024-06-03T05:04:30.827 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already out. 2024-06-03T05:04:30.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:788: destroy_osd: kill_daemons td/ceph-helpers TERM osd.0 2024-06-03T05:04:30.842 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:04:30.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:04:30.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:04:30.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:04:30.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:04:31.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:04:31.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:789: destroy_osd: ceph osd down osd.0 2024-06-03T05:04:31.643 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already down. 2024-06-03T05:04:31.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:790: destroy_osd: ceph osd purge osd.0 --yes-i-really-mean-it 2024-06-03T05:04:32.228 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 does not exist 2024-06-03T05:04:32.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:791: destroy_osd: teardown td/ceph-helpers/0 2024-06-03T05:04:32.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers/0 2024-06-03T05:04:32.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:04:32.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers/0 KILL 2024-06-03T05:04:32.247 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:04:32.247 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:04:32.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:04:32.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:04:32.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:04:32.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:04:32.250 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:04:32.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:04:32.251 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:04:32.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:04:32.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:04:32.253 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:04:32.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:04:32.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:04:32.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:04:32.254 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:04:32.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:04:32.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:04:32.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers/0 2024-06-03T05:04:32.261 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:04:32.261 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:04:32.261 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:04:32.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:04:32.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:04:32.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:04:32.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:792: destroy_osd: rm -fr td/ceph-helpers/0 2024-06-03T05:04:32.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:803: test_destroy_osd: ceph osd dump 2024-06-03T05:04:32.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:803: test_destroy_osd: grep 'osd. ' 2024-06-03T05:04:32.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:804: test_destroy_osd: teardown td/ceph-helpers 2024-06-03T05:04:32.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:04:32.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:04:32.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:04:32.680 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:04:32.680 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:04:32.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:04:32.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:04:32.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:04:32.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:04:32.789 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:04:32.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:04:32.790 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:04:32.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:04:32.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:04:32.791 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:04:32.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:04:32.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:04:32.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:04:32.793 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:04:32.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:04:32.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:04:32.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:04:32.798 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:04:32.798 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:04:32.798 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:04:32.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:04:32.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:04:32.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:04:32.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:04:32.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_display_logs td/ceph-helpers 2024-06-03T05:04:32.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: test_display_logs: local dir=td/ceph-helpers 2024-06-03T05:04:32.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2104: test_display_logs: setup td/ceph-helpers 2024-06-03T05:04:32.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-03T05:04:32.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-03T05:04:32.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:04:32.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:04:32.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:04:32.802 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:04:32.802 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:04:32.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:04:32.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:04:32.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:04:32.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:04:32.803 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:04:32.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:04:32.804 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:04:32.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:04:32.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:04:32.805 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:04:32.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:04:32.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:04:32.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:04:32.807 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:04:32.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:04:32.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:04:32.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:04:32.810 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:04:32.810 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:04:32.810 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:04:32.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:04:32.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:04:32.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:04:32.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-03T05:04:32.812 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:04:32.812 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:04:32.812 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:04:32.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70897 2024-06-03T05:04:32.814 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:04:32.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:04:32.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:04:32.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-03T05:04:32.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2105: test_display_logs: run_mon td/ceph-helpers a 2024-06-03T05:04:32.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-03T05:04:32.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:04:32.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:04:32.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:04:32.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-03T05:04:32.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-06-03T05:04:33.238 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:04:33.238 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:04:33.238 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:04:33.238 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:04:33.238 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:04:33.238 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:04:33.238 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:04:33.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-03T05:04:33.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:04:33.467 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:04:33.467 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:04:33.467 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:04:33.467 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:04:33.467 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:04:33.467 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:04:33.467 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:04:33.468 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:04:33.468 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:04:33.468 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:04:33.468 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:04:33.469 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:04:33.469 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:04:33.469 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get fsid 2024-06-03T05:04:33.579 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:04:33.579 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:04:33.579 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:04:33.579 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:04:33.580 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:04:33.580 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:04:33.580 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:04:33.580 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:04:33.581 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:04:33.581 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:04:33.581 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:04:33.581 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:04:33.582 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:04:33.582 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get mon_host 2024-06-03T05:04:33.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2106: test_display_logs: kill_daemons td/ceph-helpers 2024-06-03T05:04:33.690 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:04:33.691 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:04:33.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:04:33.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:04:33.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:04:35.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:04:35.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2107: test_display_logs: display_logs td/ceph-helpers 2024-06-03T05:04:35.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2092: display_logs: local dir=td/ceph-helpers 2024-06-03T05:04:35.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2094: display_logs: find td/ceph-helpers -maxdepth 1 -name '*.log' 2024-06-03T05:04:35.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2095: display_logs: read file 2024-06-03T05:04:35.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2096: display_logs: echo '======================= td/ceph-helpers/mon.a.log' 2024-06-03T05:04:35.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2097: display_logs: cat td/ceph-helpers/mon.a.log 2024-06-03T05:04:35.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2095: display_logs: read file 2024-06-03T05:04:35.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2108: test_display_logs: grep --quiet mon.a.log td/ceph-helpers/log.out 2024-06-03T05:04:35.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2109: test_display_logs: teardown td/ceph-helpers 2024-06-03T05:04:35.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:04:35.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:04:35.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:04:35.008 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:04:35.009 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:04:35.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:04:35.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:04:35.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:04:35.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:04:35.013 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:04:35.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:04:35.014 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:04:35.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:04:35.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:04:35.016 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:04:35.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:04:35.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:04:35.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:04:35.018 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:04:35.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:04:35.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:04:35.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:04:35.022 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:04:35.022 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:04:35.023 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:04:35.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:04:35.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:04:35.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:04:35.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:04:35.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_erasure_code_plugin_exists td/ceph-helpers 2024-06-03T05:04:35.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: test_erasure_code_plugin_exists: local dir=td/ceph-helpers 2024-06-03T05:04:35.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: test_erasure_code_plugin_exists: setup td/ceph-helpers 2024-06-03T05:04:35.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-03T05:04:35.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-03T05:04:35.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:04:35.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:04:35.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:04:35.026 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:04:35.026 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:04:35.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:04:35.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:04:35.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:04:35.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:04:35.028 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:04:35.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:04:35.029 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:04:35.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:04:35.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:04:35.031 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:04:35.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:04:35.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:04:35.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:04:35.033 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:04:35.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:04:35.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:04:35.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:04:35.037 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:04:35.037 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:04:35.037 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:04:35.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:04:35.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:04:35.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:04:35.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-03T05:04:35.040 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:04:35.040 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:04:35.040 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:04:35.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70897 2024-06-03T05:04:35.042 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:04:35.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:04:35.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:04:35.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-03T05:04:35.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: test_erasure_code_plugin_exists: run_mon td/ceph-helpers a 2024-06-03T05:04:35.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-03T05:04:35.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:04:35.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:04:35.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:04:35.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-03T05:04:35.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-06-03T05:04:35.469 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:04:35.469 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:04:35.469 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:04:35.470 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:04:35.470 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:04:35.470 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:04:35.470 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:04:35.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-03T05:04:35.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:04:35.682 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:04:35.682 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:04:35.682 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:04:35.683 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:04:35.683 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:04:35.684 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:04:35.684 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:04:35.684 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:04:35.684 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:04:35.684 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:04:35.684 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:04:35.684 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:04:35.684 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:04:35.684 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get fsid 2024-06-03T05:04:35.799 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:04:35.799 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:04:35.799 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:04:35.799 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:04:35.800 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:04:35.800 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:04:35.800 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:04:35.800 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:04:35.801 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:04:35.801 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:04:35.801 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:04:35.801 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:04:35.801 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:04:35.801 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get mon_host 2024-06-03T05:04:35.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: test_erasure_code_plugin_exists: run_mgr td/ceph-helpers x 2024-06-03T05:04:35.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-03T05:04:35.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T05:04:35.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T05:04:35.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T05:04:35.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-03T05:04:35.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T05:04:36.828 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T05:04:36.828 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:04:36.828 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:04:36.829 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:04:36.829 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:04:36.829 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:04:36.829 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:04:36.830 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:04:36.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:04:36.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: test_erasure_code_plugin_exists: erasure_code_plugin_exists jerasure 2024-06-03T05:04:36.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2051: erasure_code_plugin_exists: local plugin=jerasure 2024-06-03T05:04:36.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2052: erasure_code_plugin_exists: local status 2024-06-03T05:04:36.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2053: erasure_code_plugin_exists: local grepstr 2024-06-03T05:04:36.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2054: erasure_code_plugin_exists: local s 2024-06-03T05:04:36.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2055: erasure_code_plugin_exists: case `uname` in 2024-06-03T05:04:36.869 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2055: erasure_code_plugin_exists: uname 2024-06-03T05:04:36.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2057: erasure_code_plugin_exists: grepstr='jerasure.*No such file' 2024-06-03T05:04:36.870 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2060: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=jerasure 2024-06-03T05:04:37.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2060: erasure_code_plugin_exists: s= 2024-06-03T05:04:37.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2061: erasure_code_plugin_exists: local status=0 2024-06-03T05:04:37.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2062: erasure_code_plugin_exists: '[' 0 -eq 0 ']' 2024-06-03T05:04:37.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: erasure_code_plugin_exists: ceph osd erasure-code-profile rm TESTPROFILE 2024-06-03T05:04:37.636 INFO:tasks.workunit.client.0.smithi088.stderr:erasure-code-profile TESTPROFILE does not exist 2024-06-03T05:04:37.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: erasure_code_plugin_exists: return 0 2024-06-03T05:04:37.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: test_erasure_code_plugin_exists: erasure_code_plugin_exists FAKE 2024-06-03T05:04:37.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2051: erasure_code_plugin_exists: local plugin=FAKE 2024-06-03T05:04:37.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2052: erasure_code_plugin_exists: local status 2024-06-03T05:04:37.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2053: erasure_code_plugin_exists: local grepstr 2024-06-03T05:04:37.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2054: erasure_code_plugin_exists: local s 2024-06-03T05:04:37.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2055: erasure_code_plugin_exists: case `uname` in 2024-06-03T05:04:37.654 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2055: erasure_code_plugin_exists: uname 2024-06-03T05:04:37.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2057: erasure_code_plugin_exists: grepstr='FAKE.*No such file' 2024-06-03T05:04:37.655 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2060: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=FAKE 2024-06-03T05:04:37.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2060: erasure_code_plugin_exists: s='Error EIO: load dlopen(/usr/lib/ceph/erasure-code/libec_FAKE.so): /usr/lib/ceph/erasure-code/libec_FAKE.so: cannot open shared object file: No such file or directory' 2024-06-03T05:04:37.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2061: erasure_code_plugin_exists: local status=5 2024-06-03T05:04:37.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2062: erasure_code_plugin_exists: '[' 5 -eq 0 ']' 2024-06-03T05:04:37.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: erasure_code_plugin_exists: echo Error EIO: load 'dlopen(/usr/lib/ceph/erasure-code/libec_FAKE.so):' /usr/lib/ceph/erasure-code/libec_FAKE.so: cannot open shared object file: No such file or directory 2024-06-03T05:04:37.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: erasure_code_plugin_exists: grep --quiet 'FAKE.*No such file' 2024-06-03T05:04:37.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: erasure_code_plugin_exists: return 5 2024-06-03T05:04:37.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: test_erasure_code_plugin_exists: teardown td/ceph-helpers 2024-06-03T05:04:37.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:04:37.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:04:37.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:04:37.895 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:04:37.895 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:04:37.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:04:37.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:04:37.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:04:38.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:04:38.001 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:04:38.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:04:38.003 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:04:38.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:04:38.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:04:38.005 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:04:38.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:04:38.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:04:38.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:04:38.006 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:04:38.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:04:38.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:04:38.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:04:38.011 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:04:38.011 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:04:38.011 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:04:38.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:04:38.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:04:38.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:04:38.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:04:38.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_expect_failure td/ceph-helpers 2024-06-03T05:04:38.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: test_expect_failure: local dir=td/ceph-helpers 2024-06-03T05:04:38.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: test_expect_failure: setup td/ceph-helpers 2024-06-03T05:04:38.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-03T05:04:38.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-03T05:04:38.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:04:38.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:04:38.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:04:38.014 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:04:38.014 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:04:38.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:04:38.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:04:38.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:04:38.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:04:38.015 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:04:38.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:04:38.016 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:04:38.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:04:38.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:04:38.017 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:04:38.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:04:38.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:04:38.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:04:38.019 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:04:38.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:04:38.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:04:38.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:04:38.022 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:04:38.022 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:04:38.022 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:04:38.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:04:38.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:04:38.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:04:38.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-03T05:04:38.025 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:04:38.025 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:04:38.025 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:04:38.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70897 2024-06-03T05:04:38.027 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:04:38.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:04:38.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:04:38.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-03T05:04:38.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo FAIL ; exit 1' 2024-06-03T05:04:38.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1962: expect_failure: local dir=td/ceph-helpers 2024-06-03T05:04:38.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1963: expect_failure: shift 2024-06-03T05:04:38.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1964: expect_failure: local expected=FAIL 2024-06-03T05:04:38.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1965: expect_failure: shift 2024-06-03T05:04:38.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1966: expect_failure: local success 2024-06-03T05:04:38.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: expect_failure: bash -c 'echo FAIL ; exit 1' 2024-06-03T05:04:38.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: success=false 2024-06-03T05:04:38.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: false 2024-06-03T05:04:38.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-06-03T05:04:38.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: return 0 2024-06-03T05:04:38.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo FAIL ; exit 0' 2024-06-03T05:04:38.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1962: expect_failure: local dir=td/ceph-helpers 2024-06-03T05:04:38.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1963: expect_failure: shift 2024-06-03T05:04:38.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1964: expect_failure: local expected=FAIL 2024-06-03T05:04:38.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1965: expect_failure: shift 2024-06-03T05:04:38.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1966: expect_failure: local success 2024-06-03T05:04:38.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: expect_failure: bash -c 'echo FAIL ; exit 0' 2024-06-03T05:04:38.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: expect_failure: success=true 2024-06-03T05:04:38.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: true 2024-06-03T05:04:38.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1975: expect_failure: cat td/ceph-helpers/out 2024-06-03T05:04:38.032 INFO:tasks.workunit.client.0.smithi088.stderr:FAIL 2024-06-03T05:04:38.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1976: expect_failure: return 1 2024-06-03T05:04:38.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: test_expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-06-03T05:04:38.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo UNEXPECTED ; exit 1' 2024-06-03T05:04:38.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1962: expect_failure: local dir=td/ceph-helpers 2024-06-03T05:04:38.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1963: expect_failure: shift 2024-06-03T05:04:38.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1964: expect_failure: local expected=FAIL 2024-06-03T05:04:38.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1965: expect_failure: shift 2024-06-03T05:04:38.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1966: expect_failure: local success 2024-06-03T05:04:38.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: expect_failure: bash -c 'echo UNEXPECTED ; exit 1' 2024-06-03T05:04:38.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: success=false 2024-06-03T05:04:38.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: false 2024-06-03T05:04:38.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-06-03T05:04:38.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1975: expect_failure: cat td/ceph-helpers/out 2024-06-03T05:04:38.037 INFO:tasks.workunit.client.0.smithi088.stderr:UNEXPECTED 2024-06-03T05:04:38.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1976: expect_failure: return 1 2024-06-03T05:04:38.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: test_expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-06-03T05:04:38.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1993: test_expect_failure: teardown td/ceph-helpers 2024-06-03T05:04:38.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:04:38.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:04:38.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:04:38.039 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:04:38.039 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:04:38.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:04:38.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:04:38.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:04:38.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:04:38.042 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:04:38.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:04:38.043 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:04:38.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:04:38.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:04:38.044 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:04:38.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:04:38.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:04:38.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:04:38.046 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:04:38.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:04:38.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:04:38.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:04:38.049 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:04:38.049 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:04:38.049 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:04:38.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:04:38.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:04:38.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:04:38.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:04:38.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_flush_pg_stats td/ceph-helpers 2024-06-03T05:04:38.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2225: test_flush_pg_stats: local dir=td/ceph-helpers 2024-06-03T05:04:38.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: test_flush_pg_stats: setup td/ceph-helpers 2024-06-03T05:04:38.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-03T05:04:38.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-03T05:04:38.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:04:38.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:04:38.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:04:38.052 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:04:38.052 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:04:38.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:04:38.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:04:38.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:04:38.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:04:38.053 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:04:38.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:04:38.055 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:04:38.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:04:38.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:04:38.056 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:04:38.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:04:38.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:04:38.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:04:38.057 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:04:38.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:04:38.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:04:38.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:04:38.061 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:04:38.061 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:04:38.061 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:04:38.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:04:38.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:04:38.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:04:38.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-03T05:04:38.063 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:04:38.064 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:04:38.064 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:04:38.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70897 2024-06-03T05:04:38.066 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:04:38.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:04:38.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:04:38.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-03T05:04:38.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: test_flush_pg_stats: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:04:38.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-03T05:04:38.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:04:38.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:04:38.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:04:38.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-03T05:04:38.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:04:38.480 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:04:38.480 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:04:38.481 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:04:38.481 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:04:38.481 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:04:38.481 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:04:38.481 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:04:38.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:04:38.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:04:38.693 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:04:38.694 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:04:38.694 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:04:38.694 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:04:38.694 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:04:38.695 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:04:38.695 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:04:38.695 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:04:38.695 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:04:38.695 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:04:38.695 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:04:38.696 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:04:38.696 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:04:38.696 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get fsid 2024-06-03T05:04:38.807 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:04:38.807 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:04:38.807 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:04:38.807 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:04:38.807 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:04:38.807 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:04:38.808 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:04:38.808 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:04:38.808 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:04:38.808 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:04:38.809 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:04:38.809 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:04:38.809 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:04:38.809 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get mon_host 2024-06-03T05:04:38.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: test_flush_pg_stats: run_mgr td/ceph-helpers x 2024-06-03T05:04:38.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-03T05:04:38.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T05:04:38.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T05:04:38.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T05:04:38.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-03T05:04:38.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T05:04:39.831 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T05:04:39.831 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:04:39.831 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:04:39.831 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:04:39.831 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:04:39.831 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:04:39.832 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:04:39.832 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:04:39.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:04:39.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: test_flush_pg_stats: run_osd td/ceph-helpers 0 2024-06-03T05:04:39.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:04:39.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:04:39.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T05:04:39.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:04:39.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-03T05:04:39.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:04:39.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:04:39.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:04:39.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:04:39.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-03T05:04:39.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-03T05:04:39.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:04:39.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:04:39.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:04:39.872 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:04:39.872 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:04:39.872 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:04:39.872 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:04:39.872 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:04:39.872 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:04:39.872 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:04:39.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:04:39.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:04:39.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:04:39.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:04:39.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:04:39.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:04:39.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:04:39.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:04:39.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:04:39.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:04:39.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:04:39.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:04:39.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-03T05:04:39.873 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:04:39.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=adeb0928-38b6-4915-bc8a-f39a09b4ab9a 2024-06-03T05:04:39.874 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 adeb0928-38b6-4915-bc8a-f39a09b4ab9a 2024-06-03T05:04:39.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 adeb0928-38b6-4915-bc8a-f39a09b4ab9a' 2024-06-03T05:04:39.874 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:04:39.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDnTl1mLdj7NBAApDri3r0kZscC3RlfJHyUwg== 2024-06-03T05:04:39.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDnTl1mLdj7NBAApDri3r0kZscC3RlfJHyUwg=="}' 2024-06-03T05:04:39.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new adeb0928-38b6-4915-bc8a-f39a09b4ab9a -i td/ceph-helpers/0/new.json 2024-06-03T05:04:40.180 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:04:40.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-03T05:04:40.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDnTl1mLdj7NBAApDri3r0kZscC3RlfJHyUwg== --osd-uuid adeb0928-38b6-4915-bc8a-f39a09b4ab9a 2024-06-03T05:04:40.224 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:04:40.219+0000 7f60b3ba07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:04:40.257 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:04:40.255+0000 7f60b3ba07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:04:40.259 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:04:40.255+0000 7f60b3ba07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:04:40.260 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:04:40.255+0000 7f60b3ba07c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:04:40.260 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:04:40.255+0000 7f60b3ba07c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:04:40.260 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:04:40.255+0000 7f60b3ba07c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:04:40.305 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:04:40.303+0000 7f60b3ba07c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-03T05:04:46.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-03T05:04:46.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:04:46.633 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T05:04:46.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T05:04:46.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:04:47.186 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:04:47.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T05:04:47.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:04:47.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:04:47.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:04:47.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:04:47.217 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:04:47.215+0000 7f9e0c6637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:04:47.241 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:04:47.239+0000 7f9e0c6637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:04:47.242 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:04:47.239+0000 7f9e0c6637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:04:47.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T05:04:47.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:04:47.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:04:47.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:04:47.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:04:47.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:04:47.613 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:04:47.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:04:47.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:04:47.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:04:47.811 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:04:47.807+0000 7f9e0c6637c0 -1 Falling back to public interface 2024-06-03T05:04:48.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:04:49.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:04:49.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:04:49.015 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:04:49.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:04:49.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:04:49.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:04:49.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:04:50.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:04:50.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:04:50.422 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:04:50.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:04:50.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:04:50.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:04:50.423 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:04:50.419+0000 7f9e0c6637c0 -1 osd.0 0 log_to_monitors true 2024-06-03T05:04:51.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:04:52.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:04:52.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:04:52.053 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:04:52.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:04:52.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:04:52.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:04:52.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:04:53.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:04:53.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:04:53.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:04:53.457 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:04:53.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:04:53.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:04:53.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:04:54.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:04:54.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:04:54.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:04:54.922 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:04:54.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:04:54.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:04:55.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:04:56.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:04:56.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:04:56.361 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:04:56.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:04:56.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:04:56.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:04:56.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:04:57.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:04:57.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:04:57.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:04:57.790 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:04:57.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:04:57.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:04:58.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:04:59.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:04:59.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:04:59.359 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:04:59.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:04:59.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:04:59.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:04:59.779 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:05:00.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:05:00.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:05:00.781 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:05:00.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:05:00.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:05:00.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:05:01.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:05:02.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:05:02.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:05:02.293 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:05:02.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:05:02.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:05:02.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:05:02.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:05:03.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:05:03.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:05:03.784 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:05:03.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:05:03.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:05:03.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:05:04.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:05:05.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:05:05.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:05:05.808 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:05:05.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:05:05.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:05:05.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:05:06.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:05:07.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:05:07.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:05:07.603 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:05:07.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:05:07.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:05:07.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:05:08.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:05:09.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:05:09.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:05:09.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:05:09.246 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:05:09.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:05:09.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:05:10.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:05:11.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:05:11.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:05:11.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:05:11.058 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:05:11.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:05:11.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:05:11.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:05:12.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:05:12.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:05:12.912 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:05:12.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:05:12.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:05:12.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:05:13.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:05:14.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:05:14.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:05:14.743 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:05:14.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:05:14.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:05:14.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:05:15.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:05:16.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:05:16.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:05:16.303 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-06-03T05:05:16.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T05:05:16.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:05:16.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:05:16.771 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2560644841,v1:127.0.0.1:6803/2560644841] [v2:127.0.0.1:6804/2560644841,v1:127.0.0.1:6805/2560644841] exists,up adeb0928-38b6-4915-bc8a-f39a09b4ab9a 2024-06-03T05:05:16.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:05:16.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:05:16.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:05:16.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: test_flush_pg_stats: create_rbd_pool 2024-06-03T05:05:16.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-03T05:05:17.192 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' does not exist 2024-06-03T05:05:17.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-03T05:05:17.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-03T05:05:17.743 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' already exists 2024-06-03T05:05:17.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T05:05:18.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-03T05:05:19.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: test_flush_pg_stats: rados -p rbd put obj /etc/group 2024-06-03T05:05:19.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: test_flush_pg_stats: flush_pg_stats 2024-06-03T05:05:19.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T05:05:19.957 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T05:05:20.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-06-03T05:05:20.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T05:05:20.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:05:20.379 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T05:05:20.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836487 2024-06-03T05:05:20.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836487 2024-06-03T05:05:20.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836487' 2024-06-03T05:05:20.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:05:20.549 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836487 2024-06-03T05:05:20.549 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:05:20.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T05:05:20.551 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836487 2024-06-03T05:05:20.551 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:05:20.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836487 2024-06-03T05:05:20.553 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836487 2024-06-03T05:05:20.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-06-03T05:05:20.553 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:05:20.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836485 -lt 21474836487 2024-06-03T05:05:20.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T05:05:21.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T05:05:21.950 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:05:22.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836485 -lt 21474836487 2024-06-03T05:05:22.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T05:05:23.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-03T05:05:23.362 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:05:23.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836488 -lt 21474836487 2024-06-03T05:05:23.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: test_flush_pg_stats: local 'jq_filter=.pools | .[] | select(.name == "rbd") | .stats' 2024-06-03T05:05:23.787 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: test_flush_pg_stats: ceph df detail --format=json 2024-06-03T05:05:23.788 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: test_flush_pg_stats: jq '.pools | .[] | select(.name == "rbd") | .stats.stored' 2024-06-03T05:05:24.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: test_flush_pg_stats: stored=6543 2024-06-03T05:05:24.355 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: test_flush_pg_stats: ceph df detail --format=json 2024-06-03T05:05:24.355 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: test_flush_pg_stats: jq '.pools | .[] | select(.name == "rbd") | .stats.stored_raw' 2024-06-03T05:05:24.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: test_flush_pg_stats: stored_raw=6543 2024-06-03T05:05:24.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: test_flush_pg_stats: test 6543 -gt 0 2024-06-03T05:05:24.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: test_flush_pg_stats: test 6543 == 6543 2024-06-03T05:05:24.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: test_flush_pg_stats: teardown td/ceph-helpers 2024-06-03T05:05:24.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:05:24.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:05:24.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:05:24.934 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:05:24.934 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:05:24.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:05:24.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:05:24.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:05:25.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:05:25.041 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:05:25.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:05:25.042 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:05:25.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:05:25.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:05:25.047 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:05:25.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:05:25.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:05:25.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:05:25.048 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:05:25.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:05:25.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:05:25.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:05:25.052 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:05:25.052 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:05:25.052 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:05:25.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:05:25.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:05:25.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:05:25.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:05:25.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_get_config td/ceph-helpers 2024-06-03T05:05:25.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: test_get_config: local dir=td/ceph-helpers 2024-06-03T05:05:25.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: test_get_config: setup td/ceph-helpers 2024-06-03T05:05:25.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-03T05:05:25.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-03T05:05:25.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:05:25.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:05:25.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:05:25.055 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:05:25.055 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:05:25.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:05:25.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:05:25.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:05:25.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:05:25.056 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:05:25.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:05:25.057 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:05:25.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:05:25.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:05:25.058 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:05:25.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:05:25.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:05:25.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:05:25.060 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:05:25.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:05:25.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:05:25.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:05:25.062 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:05:25.062 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:05:25.062 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:05:25.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:05:25.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:05:25.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:05:25.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-03T05:05:25.066 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:05:25.066 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:05:25.066 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:05:25.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70897 2024-06-03T05:05:25.068 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:05:25.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:05:25.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:05:25.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-03T05:05:25.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1140: test_get_config: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:05:25.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-03T05:05:25.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:05:25.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:05:25.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:05:25.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-03T05:05:25.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:05:25.448 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:05:25.448 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:05:25.448 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:05:25.448 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:05:25.448 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:05:25.448 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:05:25.448 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:05:25.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:05:25.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:05:25.674 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:05:25.674 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:05:25.674 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:05:25.675 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:05:25.676 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:05:25.676 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:05:25.676 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:05:25.676 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:05:25.676 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:05:25.676 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:05:25.676 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:05:25.676 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:05:25.676 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:05:25.676 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get fsid 2024-06-03T05:05:25.790 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:05:25.791 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:05:25.791 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:05:25.791 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:05:25.791 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:05:25.791 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:05:25.792 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:05:25.792 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:05:25.792 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:05:25.792 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:05:25.792 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:05:25.792 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:05:25.793 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:05:25.793 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get mon_host 2024-06-03T05:05:25.894 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: test_get_config: get_config mon a osd_pool_default_size 2024-06-03T05:05:25.894 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:05:25.894 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:05:25.894 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_pool_default_size 2024-06-03T05:05:25.894 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:05:25.894 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_pool_default_size 2024-06-03T05:05:25.894 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:05:25.894 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:05:25.895 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:05:25.895 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:05:25.895 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:05:25.896 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:05:25.896 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:05:25.896 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get osd_pool_default_size 2024-06-03T05:05:25.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: test_get_config: test 1 = 1 2024-06-03T05:05:25.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: test_get_config: run_mgr td/ceph-helpers x 2024-06-03T05:05:25.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-03T05:05:25.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T05:05:25.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T05:05:25.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T05:05:25.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-03T05:05:25.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T05:05:26.907 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T05:05:26.907 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:05:26.907 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:05:26.907 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:05:26.907 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:05:26.907 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:05:26.907 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:05:26.907 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:05:26.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:05:26.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1143: test_get_config: run_osd td/ceph-helpers 0 --osd_max_scrubs=3 2024-06-03T05:05:26.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:05:26.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:05:26.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T05:05:26.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:05:26.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-03T05:05:26.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:05:26.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:05:26.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:05:26.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:05:26.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-03T05:05:26.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-03T05:05:26.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:05:26.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:05:26.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:05:26.942 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:05:26.942 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:05:26.942 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:05:26.942 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:05:26.943 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:05:26.943 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:05:26.943 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:05:26.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:05:26.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:05:26.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:05:26.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:05:26.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:05:26.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:05:26.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:05:26.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:05:26.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:05:26.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:05:26.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:05:26.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_max_scrubs=3 2024-06-03T05:05:26.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-03T05:05:26.944 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:05:26.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=533f37b5-e1d6-45ca-be0f-0112ec0c3fed 2024-06-03T05:05:26.944 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 533f37b5-e1d6-45ca-be0f-0112ec0c3fed 2024-06-03T05:05:26.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 533f37b5-e1d6-45ca-be0f-0112ec0c3fed' 2024-06-03T05:05:26.945 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:05:26.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAWT11mocRCORAAExWwXCERNJzrGmHJEheHhw== 2024-06-03T05:05:26.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAWT11mocRCORAAExWwXCERNJzrGmHJEheHhw=="}' 2024-06-03T05:05:26.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 533f37b5-e1d6-45ca-be0f-0112ec0c3fed -i td/ceph-helpers/0/new.json 2024-06-03T05:05:27.297 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:05:27.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-03T05:05:27.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_max_scrubs=3 --mkfs --key AQAWT11mocRCORAAExWwXCERNJzrGmHJEheHhw== --osd-uuid 533f37b5-e1d6-45ca-be0f-0112ec0c3fed 2024-06-03T05:05:27.353 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:05:27.351+0000 7f2be64907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:05:27.374 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:05:27.371+0000 7f2be64907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:05:27.375 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:05:27.371+0000 7f2be64907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:05:27.376 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:05:27.371+0000 7f2be64907c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:05:27.376 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:05:27.371+0000 7f2be64907c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:05:27.376 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:05:27.371+0000 7f2be64907c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:05:27.406 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:05:27.403+0000 7f2be64907c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-03T05:05:33.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-03T05:05:33.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:05:33.979 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T05:05:33.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T05:05:33.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:05:34.517 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:05:34.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T05:05:34.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_max_scrubs=3 2024-06-03T05:05:34.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:05:34.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:05:34.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:05:34.543 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:05:34.539+0000 7fe2824bb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:05:34.545 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:05:34.543+0000 7fe2824bb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:05:34.546 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:05:34.543+0000 7fe2824bb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:05:34.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T05:05:34.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:05:34.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:05:34.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:05:34.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:05:34.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:05:34.958 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:05:34.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:05:34.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:05:34.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:05:35.094 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:05:35.091+0000 7fe2824bb7c0 -1 Falling back to public interface 2024-06-03T05:05:35.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:05:36.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:05:36.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:05:36.412 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:05:36.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:05:36.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:05:36.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:05:36.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:05:37.699 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:05:37.695+0000 7fe2824bb7c0 -1 osd.0 0 log_to_monitors true 2024-06-03T05:05:37.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:05:37.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:05:37.819 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:05:37.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:05:37.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:05:37.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:05:38.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:05:39.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:05:39.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:05:39.461 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:05:39.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:05:39.462 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:05:39.462 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:05:39.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:05:40.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:05:40.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:05:40.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:05:40.890 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:05:40.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:05:40.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:05:41.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:05:42.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:05:42.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:05:42.305 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:05:42.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:05:42.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:05:42.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:05:42.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:05:43.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:05:43.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:05:43.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:05:43.907 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:05:43.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:05:43.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:05:44.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:05:45.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:05:45.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:05:45.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:05:45.372 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:05:45.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:05:45.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:05:45.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:05:46.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:05:46.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:05:46.924 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:05:46.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:05:46.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:05:46.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:05:47.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:05:48.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:05:48.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:05:48.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:05:48.395 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:05:48.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:05:48.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:05:48.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:05:49.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:05:49.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:05:49.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:05:49.923 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:05:49.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:05:49.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:05:50.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:05:51.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:05:51.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:05:51.366 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:05:51.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:05:51.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:05:51.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:05:52.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:05:53.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:05:53.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:05:53.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:05:53.406 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:05:53.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:05:53.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:05:54.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:05:55.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:05:55.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:05:55.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:05:55.223 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:05:55.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:05:55.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:05:56.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:05:57.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:05:57.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:05:57.090 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:05:57.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:05:57.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:05:57.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:05:57.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:05:58.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:05:58.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:05:58.950 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:05:58.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:05:58.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:05:58.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:05:59.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:06:00.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:06:00.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:06:00.568 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:06:00.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:06:00.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:06:00.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:06:01.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:06:02.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:06:02.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:06:02.386 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:06:02.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:06:02.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:06:02.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:06:02.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:06:03.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:06:03.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:06:03.811 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-06-03T05:06:03.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T05:06:03.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:06:03.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:06:04.264 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1644366219,v1:127.0.0.1:6803/1644366219] [v2:127.0.0.1:6804/1644366219,v1:127.0.0.1:6805/1644366219] exists,up 533f37b5-e1d6-45ca-be0f-0112ec0c3fed 2024-06-03T05:06:04.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:06:04.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:06:04.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:06:04.265 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1144: test_get_config: get_config osd 0 osd_max_scrubs 2024-06-03T05:06:04.265 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=osd 2024-06-03T05:06:04.265 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=0 2024-06-03T05:06:04.265 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_max_scrubs 2024-06-03T05:06:04.267 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_max_scrubs 2024-06-03T05:06:04.267 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path osd.0 2024-06-03T05:06:04.267 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T05:06:04.267 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T05:06:04.267 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:06:04.267 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:06:04.267 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:06:04.267 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-osd.0.asok 2024-06-03T05:06:04.267 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:06:04.267 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-osd.0.asok config get osd_max_scrubs 2024-06-03T05:06:04.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1144: test_get_config: test 3 = 3 2024-06-03T05:06:04.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1145: test_get_config: teardown td/ceph-helpers 2024-06-03T05:06:04.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:06:04.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:06:04.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:06:04.373 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:06:04.373 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:06:04.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:06:04.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:06:04.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:06:04.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:06:04.482 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:06:04.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:06:04.483 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:06:04.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:06:04.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:06:04.484 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:06:04.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:06:04.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:06:04.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:06:04.485 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:06:04.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:06:04.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:06:04.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:06:04.492 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:06:04.492 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:06:04.492 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:06:04.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:06:04.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:06:04.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:06:04.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:06:04.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_get_is_making_recovery_progress td/ceph-helpers 2024-06-03T05:06:04.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: test_get_is_making_recovery_progress: local dir=td/ceph-helpers 2024-06-03T05:06:04.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: test_get_is_making_recovery_progress: setup td/ceph-helpers 2024-06-03T05:06:04.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-03T05:06:04.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-03T05:06:04.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:06:04.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:06:04.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:06:04.495 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:06:04.495 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:06:04.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:06:04.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:06:04.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:06:04.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:06:04.496 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:06:04.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:06:04.497 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:06:04.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:06:04.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:06:04.498 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:06:04.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:06:04.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:06:04.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:06:04.499 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:06:04.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:06:04.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:06:04.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:06:04.502 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:06:04.502 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:06:04.502 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:06:04.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:06:04.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:06:04.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:06:04.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-03T05:06:04.504 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:06:04.504 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:06:04.504 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:06:04.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70897 2024-06-03T05:06:04.506 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:06:04.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:06:04.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:06:04.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-03T05:06:04.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1347: test_get_is_making_recovery_progress: run_mon td/ceph-helpers a 2024-06-03T05:06:04.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-03T05:06:04.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:06:04.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:06:04.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:06:04.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-03T05:06:04.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-06-03T05:06:04.925 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:06:04.925 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:06:04.926 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:06:04.926 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:06:04.926 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:06:04.927 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:06:04.927 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:06:04.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-03T05:06:05.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:06:05.155 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:06:05.155 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:06:05.155 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:06:05.155 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:06:05.155 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:06:05.156 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:06:05.156 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:06:05.156 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:06:05.156 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:06:05.156 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:06:05.156 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:06:05.157 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:06:05.157 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:06:05.157 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get fsid 2024-06-03T05:06:05.270 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:06:05.270 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:06:05.270 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:06:05.271 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:06:05.271 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:06:05.271 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:06:05.271 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:06:05.271 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:06:05.272 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:06:05.272 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:06:05.272 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:06:05.272 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:06:05.272 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:06:05.272 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get mon_host 2024-06-03T05:06:05.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1348: test_get_is_making_recovery_progress: run_mgr td/ceph-helpers x 2024-06-03T05:06:05.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-03T05:06:05.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T05:06:05.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T05:06:05.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T05:06:05.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-03T05:06:05.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T05:06:06.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T05:06:06.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:06:06.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:06:06.276 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:06:06.276 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:06:06.276 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:06:06.276 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:06:06.276 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:06:06.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:06:06.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1349: test_get_is_making_recovery_progress: get_is_making_recovery_progress 2024-06-03T05:06:06.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T05:06:06.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T05:06:06.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T05:06:06.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T05:06:06.310 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T05:06:06.311 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T05:06:06.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-03T05:06:06.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-03T05:06:06.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1350: test_get_is_making_recovery_progress: teardown td/ceph-helpers 2024-06-03T05:06:06.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:06:06.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:06:06.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:06:06.532 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:06:06.532 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:06:06.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:06:06.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:06:06.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:06:06.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:06:06.638 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:06:06.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:06:06.639 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:06:06.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:06:06.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:06:06.640 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:06:06.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:06:06.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:06:06.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:06:06.642 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:06:06.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:06:06.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:06:06.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:06:06.646 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:06:06.646 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:06:06.646 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:06:06.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:06:06.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:06:06.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:06:06.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:06:06.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_get_last_scrub_stamp td/ceph-helpers 2024-06-03T05:06:06.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1533: test_get_last_scrub_stamp: local dir=td/ceph-helpers 2024-06-03T05:06:06.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1535: test_get_last_scrub_stamp: setup td/ceph-helpers 2024-06-03T05:06:06.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-03T05:06:06.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-03T05:06:06.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:06:06.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:06:06.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:06:06.649 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:06:06.649 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:06:06.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:06:06.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:06:06.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:06:06.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:06:06.650 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:06:06.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:06:06.651 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:06:06.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:06:06.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:06:06.653 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:06:06.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:06:06.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:06:06.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:06:06.655 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:06:06.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:06:06.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:06:06.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:06:06.658 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:06:06.658 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:06:06.658 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:06:06.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:06:06.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:06:06.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:06:06.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-03T05:06:06.660 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:06:06.660 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:06:06.660 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:06:06.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70897 2024-06-03T05:06:06.663 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:06:06.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:06:06.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:06:06.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-03T05:06:06.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1536: test_get_last_scrub_stamp: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:06:06.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-03T05:06:06.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:06:06.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:06:06.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:06:06.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-03T05:06:06.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:06:07.068 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:06:07.068 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:06:07.068 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:06:07.068 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:06:07.069 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:06:07.069 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:06:07.069 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:06:07.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:06:07.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:06:07.305 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:06:07.305 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:06:07.305 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:06:07.305 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:06:07.306 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:06:07.306 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:06:07.307 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:06:07.307 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:06:07.307 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:06:07.307 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:06:07.307 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:06:07.307 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:06:07.307 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:06:07.307 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get fsid 2024-06-03T05:06:07.420 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:06:07.420 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:06:07.420 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:06:07.420 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:06:07.421 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:06:07.421 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:06:07.421 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:06:07.422 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:06:07.422 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:06:07.422 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:06:07.422 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:06:07.422 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:06:07.422 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:06:07.422 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get mon_host 2024-06-03T05:06:07.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1537: test_get_last_scrub_stamp: run_mgr td/ceph-helpers x 2024-06-03T05:06:07.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-03T05:06:07.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T05:06:07.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T05:06:07.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T05:06:07.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-03T05:06:07.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T05:06:08.418 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T05:06:08.418 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:06:08.418 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:06:08.418 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:06:08.418 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:06:08.418 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:06:08.419 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:06:08.419 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:06:08.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:06:08.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1538: test_get_last_scrub_stamp: run_osd td/ceph-helpers 0 2024-06-03T05:06:08.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:06:08.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:06:08.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T05:06:08.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:06:08.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-03T05:06:08.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:06:08.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:06:08.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:06:08.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:06:08.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-03T05:06:08.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-03T05:06:08.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:06:08.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:06:08.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:06:08.452 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:06:08.452 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:06:08.452 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:06:08.452 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:06:08.452 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:06:08.452 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:06:08.452 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:06:08.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:06:08.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:06:08.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:06:08.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:06:08.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:06:08.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:06:08.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:06:08.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:06:08.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:06:08.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:06:08.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:06:08.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:06:08.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-03T05:06:08.453 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:06:08.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=eabdc86e-e579-4be6-991e-604ac47856ff 2024-06-03T05:06:08.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 eabdc86e-e579-4be6-991e-604ac47856ff' 2024-06-03T05:06:08.454 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 eabdc86e-e579-4be6-991e-604ac47856ff 2024-06-03T05:06:08.454 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:06:08.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBAT11mEBbwGxAAt7qgo9xU0ELs9FB8DlGruA== 2024-06-03T05:06:08.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBAT11mEBbwGxAAt7qgo9xU0ELs9FB8DlGruA=="}' 2024-06-03T05:06:08.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new eabdc86e-e579-4be6-991e-604ac47856ff -i td/ceph-helpers/0/new.json 2024-06-03T05:06:08.792 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:06:08.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-03T05:06:08.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBAT11mEBbwGxAAt7qgo9xU0ELs9FB8DlGruA== --osd-uuid eabdc86e-e579-4be6-991e-604ac47856ff 2024-06-03T05:06:08.840 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:06:08.835+0000 7f918f7627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:06:08.877 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:06:08.875+0000 7f918f7627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:06:08.878 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:06:08.875+0000 7f918f7627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:06:08.879 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:06:08.875+0000 7f918f7627c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:06:08.879 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:06:08.875+0000 7f918f7627c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:06:08.879 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:06:08.875+0000 7f918f7627c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:06:08.926 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:06:08.923+0000 7f918f7627c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-03T05:06:15.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-03T05:06:15.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:06:15.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T05:06:15.243 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T05:06:15.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:06:15.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T05:06:15.806 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:06:15.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:06:15.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:06:15.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:06:15.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:06:15.831 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:06:15.827+0000 7fbceecb97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:06:15.834 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:06:15.831+0000 7fbceecb97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:06:15.835 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:06:15.831+0000 7fbceecb97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:06:16.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T05:06:16.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:06:16.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:06:16.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:06:16.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:06:16.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:06:16.263 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:06:16.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:06:16.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:06:16.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:06:16.414 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:06:16.411+0000 7fbceecb97c0 -1 Falling back to public interface 2024-06-03T05:06:16.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:06:17.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:06:17.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:06:17.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:06:17.687 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:06:17.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:06:17.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:06:18.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:06:19.108 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:06:19.103+0000 7fbceecb97c0 -1 osd.0 0 log_to_monitors true 2024-06-03T05:06:19.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:06:19.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:06:19.155 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:06:19.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:06:19.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:06:19.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:06:19.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:06:20.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:06:20.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:06:20.828 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:06:20.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:06:20.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:06:20.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:06:21.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:06:22.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:06:22.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:06:22.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:06:22.350 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:06:22.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:06:22.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:06:22.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:06:23.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:06:23.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:06:23.828 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:06:23.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:06:23.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:06:23.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:06:24.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:06:25.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:06:25.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:06:25.250 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:06:25.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:06:25.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:06:25.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:06:25.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:06:26.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:06:26.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:06:26.713 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:06:26.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:06:26.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:06:26.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:06:27.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:06:28.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:06:28.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:06:28.158 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:06:28.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:06:28.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:06:28.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:06:28.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:06:29.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:06:29.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:06:29.580 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:06:29.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:06:29.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:06:29.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:06:30.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:06:31.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:06:31.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:06:31.155 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:06:31.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:06:31.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:06:31.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:06:31.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:06:32.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:06:32.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:06:32.601 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:06:32.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:06:32.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:06:32.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:06:33.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:06:34.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:06:34.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:06:34.520 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:06:34.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:06:34.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:06:34.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:06:35.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:06:36.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:06:36.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:06:36.316 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:06:36.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:06:36.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:06:36.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:06:37.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:06:38.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:06:38.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:06:38.172 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:06:38.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:06:38.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:06:38.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:06:39.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:06:40.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:06:40.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:06:40.011 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:06:40.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:06:40.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:06:40.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:06:41.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:06:42.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:06:42.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:06:42.140 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:06:42.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:06:42.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:06:42.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:06:43.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:06:44.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:06:44.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:06:44.183 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:06:44.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:06:44.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:06:44.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:06:44.596 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2710326758,v1:127.0.0.1:6803/2710326758] [v2:127.0.0.1:6804/2710326758,v1:127.0.0.1:6805/2710326758] exists,up eabdc86e-e579-4be6-991e-604ac47856ff 2024-06-03T05:06:44.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:06:44.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:06:44.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:06:44.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1539: test_get_last_scrub_stamp: create_rbd_pool 2024-06-03T05:06:44.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-03T05:06:44.981 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' does not exist 2024-06-03T05:06:44.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-03T05:06:44.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-03T05:06:45.456 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' already exists 2024-06-03T05:06:45.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T05:06:46.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-03T05:06:47.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1540: test_get_last_scrub_stamp: wait_for_clean 2024-06-03T05:06:47.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T05:06:47.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T05:06:47.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T05:06:47.703 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T05:06:47.704 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:06:47.704 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:06:47.704 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:06:47.704 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:06:47.704 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:06:47.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T05:06:47.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T05:06:47.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T05:06:47.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T05:06:47.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T05:06:47.824 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T05:06:48.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-06-03T05:06:48.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T05:06:48.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:06:48.249 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T05:06:48.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836489 2024-06-03T05:06:48.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836489 2024-06-03T05:06:48.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489' 2024-06-03T05:06:48.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:06:48.433 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836489 2024-06-03T05:06:48.433 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:06:48.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T05:06:48.434 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836489 2024-06-03T05:06:48.434 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:06:48.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836489 2024-06-03T05:06:48.435 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836489 2024-06-03T05:06:48.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2024-06-03T05:06:48.436 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:06:48.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836487 -lt 21474836489 2024-06-03T05:06:48.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T05:06:49.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T05:06:49.840 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:06:50.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836489 -lt 21474836489 2024-06-03T05:06:50.249 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T05:06:50.249 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:06:50.249 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:06:50.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-03T05:06:50.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T05:06:50.801 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T05:06:50.801 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T05:06:50.801 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T05:06:50.801 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T05:06:50.801 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T05:06:50.802 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T05:06:51.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T05:06:51.215 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T05:06:51.215 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:06:51.216 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:06:51.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-03T05:06:51.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T05:06:51.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T05:06:51.729 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1541: test_get_last_scrub_stamp: get_last_scrub_stamp 1.0 2024-06-03T05:06:51.729 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T05:06:51.729 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T05:06:51.729 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T05:06:51.729 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T05:06:52.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1541: test_get_last_scrub_stamp: stamp=2024-06-03T05:06:45.304740+0000 2024-06-03T05:06:52.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1542: test_get_last_scrub_stamp: test -n 2024-06-03T05:06:45.304740+0000 2024-06-03T05:06:52.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1543: test_get_last_scrub_stamp: teardown td/ceph-helpers 2024-06-03T05:06:52.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:06:52.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:06:52.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:06:52.149 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:06:52.149 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:06:52.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:06:52.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:06:52.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:06:52.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:06:52.456 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:06:52.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:06:52.457 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:06:52.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:06:52.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:06:52.459 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:06:52.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:06:52.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:06:52.461 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:06:52.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:06:52.462 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:06:52.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:06:52.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:06:52.471 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:06:52.471 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:06:52.471 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:06:52.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:06:52.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:06:52.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:06:52.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:06:52.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_get_not_primary td/ceph-helpers 2024-06-03T05:06:52.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: test_get_not_primary: local dir=td/ceph-helpers 2024-06-03T05:06:52.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: test_get_not_primary: setup td/ceph-helpers 2024-06-03T05:06:52.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-03T05:06:52.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-03T05:06:52.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:06:52.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:06:52.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:06:52.474 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:06:52.474 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:06:52.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:06:52.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:06:52.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:06:52.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:06:52.476 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:06:52.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:06:52.477 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:06:52.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:06:52.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:06:52.479 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:06:52.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:06:52.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:06:52.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:06:52.480 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:06:52.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:06:52.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:06:52.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:06:52.484 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:06:52.484 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:06:52.484 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:06:52.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:06:52.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:06:52.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:06:52.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-03T05:06:52.487 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:06:52.487 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:06:52.487 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:06:52.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70897 2024-06-03T05:06:52.489 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:06:52.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:06:52.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:06:52.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-03T05:06:52.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: test_get_not_primary: run_mon td/ceph-helpers a --osd_pool_default_size=2 2024-06-03T05:06:52.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-03T05:06:52.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:06:52.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:06:52.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:06:52.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-03T05:06:52.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2024-06-03T05:06:52.914 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:06:52.914 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:06:52.914 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:06:52.915 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:06:52.915 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:06:52.915 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:06:52.915 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:06:52.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-06-03T05:06:53.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:06:53.134 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:06:53.135 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:06:53.135 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:06:53.135 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:06:53.136 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:06:53.136 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:06:53.136 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:06:53.137 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:06:53.137 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:06:53.137 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:06:53.137 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:06:53.137 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:06:53.137 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:06:53.137 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get fsid 2024-06-03T05:06:53.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:06:53.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:06:53.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:06:53.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:06:53.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:06:53.247 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:06:53.247 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:06:53.247 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:06:53.247 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:06:53.248 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:06:53.248 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:06:53.248 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:06:53.248 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:06:53.248 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get mon_host 2024-06-03T05:06:53.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: test_get_not_primary: run_mgr td/ceph-helpers x 2024-06-03T05:06:53.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-03T05:06:53.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T05:06:53.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T05:06:53.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T05:06:53.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-03T05:06:53.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T05:06:54.245 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T05:06:54.245 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:06:54.245 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:06:54.246 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:06:54.246 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:06:54.246 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:06:54.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:06:54.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:06:54.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:06:54.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1243: test_get_not_primary: run_osd td/ceph-helpers 0 2024-06-03T05:06:54.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:06:54.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:06:54.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T05:06:54.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:06:54.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-03T05:06:54.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:06:54.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:06:54.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:06:54.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:06:54.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-03T05:06:54.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-03T05:06:54.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:06:54.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:06:54.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:06:54.282 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:06:54.282 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:06:54.282 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:06:54.282 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:06:54.283 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:06:54.283 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:06:54.283 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:06:54.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:06:54.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:06:54.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:06:54.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:06:54.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:06:54.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:06:54.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:06:54.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:06:54.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:06:54.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:06:54.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:06:54.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:06:54.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-03T05:06:54.284 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:06:54.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1333c2c7-72cc-44aa-afff-83dc52ba1f9e 2024-06-03T05:06:54.284 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 1333c2c7-72cc-44aa-afff-83dc52ba1f9e 2024-06-03T05:06:54.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1333c2c7-72cc-44aa-afff-83dc52ba1f9e' 2024-06-03T05:06:54.285 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:06:54.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBuT11mwgbSERAAo5m0HaZR8XeMDB9a+1iBlA== 2024-06-03T05:06:54.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBuT11mwgbSERAAo5m0HaZR8XeMDB9a+1iBlA=="}' 2024-06-03T05:06:54.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1333c2c7-72cc-44aa-afff-83dc52ba1f9e -i td/ceph-helpers/0/new.json 2024-06-03T05:06:54.580 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:06:54.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-03T05:06:54.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBuT11mwgbSERAAo5m0HaZR8XeMDB9a+1iBlA== --osd-uuid 1333c2c7-72cc-44aa-afff-83dc52ba1f9e 2024-06-03T05:06:54.621 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:06:54.619+0000 7f20c60707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:06:54.640 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:06:54.635+0000 7f20c60707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:06:54.641 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:06:54.639+0000 7f20c60707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:06:54.641 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:06:54.639+0000 7f20c60707c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:06:54.641 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:06:54.639+0000 7f20c60707c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:06:54.641 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:06:54.639+0000 7f20c60707c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:06:54.680 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:06:54.675+0000 7f20c60707c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-03T05:07:01.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-03T05:07:01.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:07:01.012 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T05:07:01.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T05:07:01.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:07:01.560 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:07:01.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T05:07:01.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:07:01.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:07:01.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:07:01.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:07:01.591 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:07:01.587+0000 7f38cbd247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:07:01.612 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:07:01.607+0000 7f38cbd247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:07:01.613 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:07:01.611+0000 7f38cbd247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:07:01.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T05:07:01.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:07:01.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:07:01.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:07:01.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:07:01.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:07:01.982 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:07:01.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:07:01.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:07:01.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:07:02.187 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:07:02.183+0000 7f38cbd247c0 -1 Falling back to public interface 2024-06-03T05:07:02.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:07:03.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:07:03.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:07:03.394 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:07:03.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:07:03.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:07:03.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:07:03.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:07:04.761 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:07:04.759+0000 7f38cbd247c0 -1 osd.0 0 log_to_monitors true 2024-06-03T05:07:04.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:07:04.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:07:04.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:07:04.803 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:07:04.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:07:04.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:07:05.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:07:06.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:07:06.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:07:06.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:07:06.402 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:07:06.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:07:06.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:07:06.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:07:07.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:07:07.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:07:07.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:07:07.831 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:07:07.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:07:07.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:07:08.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:07:09.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:07:09.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:07:09.291 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:07:09.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:07:09.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:07:09.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:07:09.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:07:10.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:07:10.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:07:10.874 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:07:10.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:07:10.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:07:10.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:07:11.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:07:12.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:07:12.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:07:12.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:07:12.280 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:07:12.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:07:12.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:07:12.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:07:13.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:07:13.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:07:13.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:07:13.855 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:07:13.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:07:13.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:07:14.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:07:15.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:07:15.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:07:15.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:07:15.329 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:07:15.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:07:15.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:07:15.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:07:16.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:07:16.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:07:16.931 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:07:16.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:07:16.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:07:16.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:07:17.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:07:18.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:07:18.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:07:18.345 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:07:18.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:07:18.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:07:18.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:07:19.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:07:20.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:07:20.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:07:20.054 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:07:20.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:07:20.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:07:20.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:07:21.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:07:22.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:07:22.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:07:22.135 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:07:22.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:07:22.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:07:22.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:07:22.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:07:23.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:07:23.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:07:23.963 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:07:23.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:07:23.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:07:23.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:07:24.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:07:25.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:07:25.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:07:25.681 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:07:25.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:07:25.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:07:25.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:07:26.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:07:27.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:07:27.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:07:27.531 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:07:27.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:07:27.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:07:27.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:07:28.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:07:29.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:07:29.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:07:29.572 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:07:29.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:07:29.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:07:29.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:07:29.983 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1383084624,v1:127.0.0.1:6803/1383084624] [v2:127.0.0.1:6804/1383084624,v1:127.0.0.1:6805/1383084624] exists,up 1333c2c7-72cc-44aa-afff-83dc52ba1f9e 2024-06-03T05:07:29.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:07:29.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:07:29.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:07:29.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: test_get_not_primary: run_osd td/ceph-helpers 1 2024-06-03T05:07:29.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:07:29.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:07:29.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-03T05:07:29.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:07:29.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-06-03T05:07:29.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:07:29.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:07:29.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:07:29.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:07:29.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-06-03T05:07:29.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-06-03T05:07:29.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:07:29.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:07:29.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:07:29.989 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:07:29.989 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:07:29.989 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:07:29.989 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:07:29.990 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:07:29.990 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:07:29.990 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:07:29.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:07:29.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:07:29.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:07:29.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:07:29.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:07:29.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:07:29.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:07:29.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:07:29.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:07:29.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:07:29.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:07:29.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:07:29.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-06-03T05:07:29.991 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:07:29.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bc77715f-81c0-47f7-afed-6dd3fa2e23ac 2024-06-03T05:07:29.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 bc77715f-81c0-47f7-afed-6dd3fa2e23ac' 2024-06-03T05:07:29.992 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 bc77715f-81c0-47f7-afed-6dd3fa2e23ac 2024-06-03T05:07:29.992 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:07:30.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCST11m92atABAA62iLDsaDd4qmZIUEprmvyA== 2024-06-03T05:07:30.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCST11m92atABAA62iLDsaDd4qmZIUEprmvyA=="}' 2024-06-03T05:07:30.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bc77715f-81c0-47f7-afed-6dd3fa2e23ac -i td/ceph-helpers/1/new.json 2024-06-03T05:07:30.549 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:07:30.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-06-03T05:07:30.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCST11m92atABAA62iLDsaDd4qmZIUEprmvyA== --osd-uuid bc77715f-81c0-47f7-afed-6dd3fa2e23ac 2024-06-03T05:07:30.599 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:07:30.595+0000 7fb8518667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:07:30.610 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:07:30.607+0000 7fb8518667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:07:30.611 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:07:30.607+0000 7fb8518667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:07:30.612 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:07:30.607+0000 7fb8518667c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-03T05:07:30.612 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:07:30.607+0000 7fb8518667c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-03T05:07:30.612 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:07:30.607+0000 7fb8518667c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-03T05:07:30.650 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:07:30.647+0000 7fb8518667c0 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-06-03T05:07:37.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-06-03T05:07:37.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:07:37.333 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-06-03T05:07:37.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-03T05:07:37.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:07:37.889 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-06-03T05:07:37.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-03T05:07:37.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:07:37.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:07:37.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:07:37.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:07:37.921 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:07:37.919+0000 7f6e6ad8e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:07:37.923 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:07:37.919+0000 7f6e6ad8e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:07:37.924 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:07:37.919+0000 7f6e6ad8e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:07:38.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-03T05:07:38.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:07:38.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T05:07:38.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:07:38.334 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:07:38.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:07:38.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:07:38.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:07:38.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:07:38.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:07:38.490 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:07:38.487+0000 7f6e6ad8e7c0 -1 Falling back to public interface 2024-06-03T05:07:38.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:07:39.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:07:39.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:07:39.758 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:07:39.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:07:39.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:07:39.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:07:40.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:07:41.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:07:41.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:07:41.178 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:07:41.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:07:41.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:07:41.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:07:41.240 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:07:41.235+0000 7f6e6ad8e7c0 -1 osd.1 0 log_to_monitors true 2024-06-03T05:07:41.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:07:42.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:07:42.823 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:07:42.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:07:42.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:07:42.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:07:42.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:07:43.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:07:44.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:07:44.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:07:44.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:07:44.406 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:07:44.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:07:44.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:07:44.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:07:45.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:07:45.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:07:45.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:07:45.849 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:07:45.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:07:45.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:07:46.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:07:47.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:07:47.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:07:47.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:07:47.288 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:07:47.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:07:47.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:07:47.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:07:48.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:07:48.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:07:48.845 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:07:48.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:07:48.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:07:48.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:07:49.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:07:50.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:07:50.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:07:50.338 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:07:50.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:07:50.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:07:50.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:07:50.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:07:51.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:07:51.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:07:51.789 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:07:51.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:07:51.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:07:51.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:07:52.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:07:53.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:07:53.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:07:53.236 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:07:53.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:07:53.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:07:53.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:07:53.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:07:54.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:07:54.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:07:54.746 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:07:54.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:07:54.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:07:54.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:07:55.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:07:56.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:07:56.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:07:56.667 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:07:56.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:07:56.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:07:56.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:07:57.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:07:58.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:07:58.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:07:58.451 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:07:58.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:07:58.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:07:58.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:07:59.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:08:00.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:08:00.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:08:00.485 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:08:00.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:08:00.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:08:00.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:08:01.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:08:02.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:08:02.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:08:02.328 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:08:02.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:08:02.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:08:02.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:08:03.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:08:04.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:08:04.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:08:04.279 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:08:04.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:08:04.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:08:04.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:08:05.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:08:06.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:08:06.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:08:06.207 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:08:06.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:08:06.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:08:06.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:08:06.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:08:07.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:08:07.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:08:07.802 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-06-03T05:08:07.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T05:08:07.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:08:07.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:08:08.211 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3784222099,v1:127.0.0.1:6811/3784222099] [v2:127.0.0.1:6812/3784222099,v1:127.0.0.1:6813/3784222099] exists,up bc77715f-81c0-47f7-afed-6dd3fa2e23ac 2024-06-03T05:08:08.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:08:08.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:08:08.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:08:08.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: test_get_not_primary: create_rbd_pool 2024-06-03T05:08:08.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-03T05:08:08.636 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' does not exist 2024-06-03T05:08:08.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-03T05:08:08.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-03T05:08:09.603 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' already exists 2024-06-03T05:08:09.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T05:08:10.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-03T05:08:12.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1246: test_get_not_primary: wait_for_clean 2024-06-03T05:08:12.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T05:08:12.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T05:08:12.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T05:08:12.455 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T05:08:12.456 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:08:12.456 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:08:12.456 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:08:12.457 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:08:12.457 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:08:12.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T05:08:12.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T05:08:12.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T05:08:12.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T05:08:12.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T05:08:12.577 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T05:08:13.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T05:08:13.034 INFO:tasks.workunit.client.0.smithi088.stderr:1' 2024-06-03T05:08:13.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T05:08:13.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:08:13.034 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T05:08:13.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836496 2024-06-03T05:08:13.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836496 2024-06-03T05:08:13.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836496' 2024-06-03T05:08:13.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:08:13.191 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T05:08:13.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672971 2024-06-03T05:08:13.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672971 2024-06-03T05:08:13.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836496 1-42949672971' 2024-06-03T05:08:13.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:08:13.351 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836496 2024-06-03T05:08:13.351 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:08:13.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T05:08:13.352 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836496 2024-06-03T05:08:13.352 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:08:13.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836496 2024-06-03T05:08:13.353 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836496 2024-06-03T05:08:13.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2024-06-03T05:08:13.354 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:08:13.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836494 -lt 21474836496 2024-06-03T05:08:13.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T05:08:14.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T05:08:14.766 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:08:15.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836496 2024-06-03T05:08:15.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:08:15.186 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672971 2024-06-03T05:08:15.186 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:08:15.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T05:08:15.188 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672971 2024-06-03T05:08:15.188 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:08:15.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672971 2024-06-03T05:08:15.190 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672971 2024-06-03T05:08:15.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-06-03T05:08:15.190 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T05:08:15.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672971 -lt 42949672971 2024-06-03T05:08:15.605 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T05:08:15.606 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:08:15.606 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:08:16.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-03T05:08:16.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T05:08:16.192 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T05:08:16.192 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T05:08:16.193 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T05:08:16.193 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T05:08:16.193 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T05:08:16.193 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T05:08:16.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T05:08:16.584 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T05:08:16.584 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:08:16.584 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:08:17.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-03T05:08:17.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T05:08:17.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T05:08:17.098 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1247: test_get_not_primary: get_primary rbd GROUP 2024-06-03T05:08:17.098 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-06-03T05:08:17.098 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=GROUP 2024-06-03T05:08:17.098 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd GROUP 2024-06-03T05:08:17.099 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T05:08:17.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1247: test_get_not_primary: local primary=1 2024-06-03T05:08:17.512 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: test_get_not_primary: get_not_primary rbd GROUP 2024-06-03T05:08:17.512 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2024-06-03T05:08:17.512 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=GROUP 2024-06-03T05:08:17.512 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd GROUP 2024-06-03T05:08:17.512 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-06-03T05:08:17.512 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=GROUP 2024-06-03T05:08:17.512 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T05:08:17.512 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd GROUP 2024-06-03T05:08:17.938 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-03T05:08:17.938 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd GROUP 2024-06-03T05:08:17.939 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-03T05:08:18.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: test_get_not_primary: local not_primary=0 2024-06-03T05:08:18.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: test_get_not_primary: test 0 '!=' 1 2024-06-03T05:08:18.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: test_get_not_primary: test 0 = 0 -o 0 = 1 2024-06-03T05:08:18.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: test_get_not_primary: teardown td/ceph-helpers 2024-06-03T05:08:18.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:08:18.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:08:18.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:08:18.371 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:08:18.371 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:08:18.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:08:18.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:08:18.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:08:18.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:08:18.477 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:08:18.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:08:18.479 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:08:18.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:08:18.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:08:18.480 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:08:18.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:08:18.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:08:18.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:08:18.481 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:08:18.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:08:18.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:08:18.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:08:18.495 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:08:18.495 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:08:18.495 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:08:18.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:08:18.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:08:18.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:08:18.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:08:18.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_get_num_active_clean td/ceph-helpers 2024-06-03T05:08:18.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1372: test_get_num_active_clean: local dir=td/ceph-helpers 2024-06-03T05:08:18.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1374: test_get_num_active_clean: setup td/ceph-helpers 2024-06-03T05:08:18.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-03T05:08:18.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-03T05:08:18.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:08:18.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:08:18.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:08:18.498 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:08:18.498 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:08:18.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:08:18.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:08:18.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:08:18.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:08:18.499 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:08:18.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:08:18.500 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:08:18.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:08:18.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:08:18.501 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:08:18.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:08:18.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:08:18.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:08:18.502 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:08:18.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:08:18.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:08:18.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:08:18.504 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:08:18.504 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:08:18.505 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:08:18.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:08:18.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:08:18.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:08:18.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-03T05:08:18.507 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:08:18.507 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:08:18.507 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:08:18.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70897 2024-06-03T05:08:18.508 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:08:18.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:08:18.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:08:18.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-03T05:08:18.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1375: test_get_num_active_clean: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:08:18.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-03T05:08:18.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:08:18.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:08:18.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:08:18.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-03T05:08:18.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:08:18.912 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:08:18.912 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:08:18.912 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:08:18.912 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:08:18.913 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:08:18.913 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:08:18.913 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:08:18.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:08:19.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:08:19.133 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:08:19.133 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:08:19.133 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:08:19.133 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:08:19.134 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:08:19.134 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:08:19.134 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:08:19.134 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:08:19.135 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:08:19.135 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:08:19.135 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:08:19.135 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:08:19.135 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:08:19.135 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get fsid 2024-06-03T05:08:19.252 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:08:19.252 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:08:19.252 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:08:19.252 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:08:19.252 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:08:19.252 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:08:19.253 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:08:19.253 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:08:19.253 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:08:19.253 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:08:19.253 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:08:19.254 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:08:19.254 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:08:19.254 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get mon_host 2024-06-03T05:08:19.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1376: test_get_num_active_clean: run_mgr td/ceph-helpers x 2024-06-03T05:08:19.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-03T05:08:19.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T05:08:19.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T05:08:19.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T05:08:19.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-03T05:08:19.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T05:08:20.278 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T05:08:20.279 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:08:20.279 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:08:20.279 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:08:20.279 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:08:20.279 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:08:20.280 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:08:20.280 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:08:20.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:08:20.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1377: test_get_num_active_clean: run_osd td/ceph-helpers 0 2024-06-03T05:08:20.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:08:20.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:08:20.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T05:08:20.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:08:20.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-03T05:08:20.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:08:20.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:08:20.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:08:20.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:08:20.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-03T05:08:20.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-03T05:08:20.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:08:20.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:08:20.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:08:20.317 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:08:20.317 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:08:20.317 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:08:20.317 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:08:20.317 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:08:20.317 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:08:20.317 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:08:20.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:08:20.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:08:20.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:08:20.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:08:20.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:08:20.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:08:20.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:08:20.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:08:20.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:08:20.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:08:20.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:08:20.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:08:20.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-03T05:08:20.319 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:08:20.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=05c39a49-8300-491e-a307-64d13175c15a 2024-06-03T05:08:20.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 05c39a49-8300-491e-a307-64d13175c15a' 2024-06-03T05:08:20.319 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 05c39a49-8300-491e-a307-64d13175c15a 2024-06-03T05:08:20.319 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:08:20.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDET11moHjzExAAA05xTBGKy1pKmb1UcHcOcA== 2024-06-03T05:08:20.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDET11moHjzExAAA05xTBGKy1pKmb1UcHcOcA=="}' 2024-06-03T05:08:20.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 05c39a49-8300-491e-a307-64d13175c15a -i td/ceph-helpers/0/new.json 2024-06-03T05:08:20.653 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:08:20.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-03T05:08:20.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDET11moHjzExAAA05xTBGKy1pKmb1UcHcOcA== --osd-uuid 05c39a49-8300-491e-a307-64d13175c15a 2024-06-03T05:08:20.699 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:08:20.695+0000 7fa5158f47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:08:20.729 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:08:20.727+0000 7fa5158f47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:08:20.731 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:08:20.727+0000 7fa5158f47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:08:20.732 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:08:20.727+0000 7fa5158f47c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:08:20.732 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:08:20.727+0000 7fa5158f47c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:08:20.732 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:08:20.727+0000 7fa5158f47c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:08:20.770 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:08:20.767+0000 7fa5158f47c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-03T05:08:27.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-03T05:08:27.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:08:27.092 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T05:08:27.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T05:08:27.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:08:27.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T05:08:27.654 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:08:27.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:08:27.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:08:27.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:08:27.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:08:27.685 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:08:27.683+0000 7f3f458f87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:08:27.688 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:08:27.683+0000 7f3f458f87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:08:27.689 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:08:27.687+0000 7f3f458f87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:08:28.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T05:08:28.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:08:28.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:08:28.095 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:08:28.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:08:28.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:08:28.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:08:28.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:08:28.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:08:28.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:08:28.250 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:08:28.247+0000 7f3f458f87c0 -1 Falling back to public interface 2024-06-03T05:08:28.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:08:29.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:08:29.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:08:29.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:08:29.516 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:08:29.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:08:29.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:08:29.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:08:30.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:08:30.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:08:30.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:08:30.931 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:08:30.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:08:30.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:08:30.946 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:08:30.943+0000 7f3f458f87c0 -1 osd.0 0 log_to_monitors true 2024-06-03T05:08:31.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:08:32.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:08:32.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:08:32.476 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:08:32.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:08:32.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:08:32.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:08:33.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:08:34.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:08:34.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:08:34.035 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:08:34.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:08:34.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:08:34.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:08:34.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:08:35.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:08:35.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:08:35.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:08:35.457 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:08:35.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:08:35.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:08:35.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:08:36.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:08:36.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:08:36.966 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:08:36.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:08:36.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:08:36.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:08:37.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:08:38.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:08:38.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:08:38.407 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:08:38.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:08:38.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:08:38.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:08:38.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:08:39.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:08:39.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:08:39.915 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:08:39.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:08:39.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:08:39.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:08:40.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:08:41.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:08:41.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:08:41.327 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:08:41.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:08:41.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:08:41.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:08:41.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:08:42.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:08:42.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:08:42.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:08:42.803 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:08:42.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:08:42.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:08:43.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:08:44.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:08:44.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:08:44.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:08:44.230 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:08:44.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:08:44.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:08:44.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:08:45.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:08:45.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:08:45.894 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:08:45.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:08:45.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:08:45.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:08:46.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:08:47.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:08:47.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:08:47.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:08:47.734 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:08:47.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:08:47.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:08:48.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:08:49.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:08:49.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:08:49.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:08:49.505 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:08:49.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:08:49.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:08:50.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:08:51.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:08:51.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:08:51.142 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:08:51.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:08:51.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:08:51.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:08:52.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:08:53.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:08:53.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:08:53.013 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:08:53.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:08:53.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:08:53.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:08:54.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:08:55.099 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:08:55.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:08:55.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:08:55.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:08:55.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:08:55.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:08:55.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:08:56.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:08:56.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:08:56.777 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-06-03T05:08:56.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T05:08:56.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:08:56.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:08:57.206 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3856231095,v1:127.0.0.1:6803/3856231095] [v2:127.0.0.1:6804/3856231095,v1:127.0.0.1:6805/3856231095] exists,up 05c39a49-8300-491e-a307-64d13175c15a 2024-06-03T05:08:57.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:08:57.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:08:57.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:08:57.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1378: test_get_num_active_clean: create_rbd_pool 2024-06-03T05:08:57.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-03T05:08:57.647 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' does not exist 2024-06-03T05:08:57.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-03T05:08:57.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-03T05:08:58.198 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' already exists 2024-06-03T05:08:58.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T05:08:59.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-03T05:09:00.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1379: test_get_num_active_clean: wait_for_clean 2024-06-03T05:09:00.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T05:09:00.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T05:09:00.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T05:09:00.244 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T05:09:00.244 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:09:00.244 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:09:00.245 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:09:00.245 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:09:00.245 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:09:00.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T05:09:00.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T05:09:00.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T05:09:00.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T05:09:00.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T05:09:00.365 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T05:09:00.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-06-03T05:09:00.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T05:09:00.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:09:00.790 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T05:09:00.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836488 2024-06-03T05:09:00.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836488 2024-06-03T05:09:00.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836488' 2024-06-03T05:09:00.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:09:00.975 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836488 2024-06-03T05:09:00.975 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:09:00.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T05:09:00.977 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836488 2024-06-03T05:09:00.978 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:09:00.978 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836488 2024-06-03T05:09:00.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836488 2024-06-03T05:09:00.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2024-06-03T05:09:00.979 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:09:01.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 6 -lt 21474836488 2024-06-03T05:09:01.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T05:09:02.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T05:09:02.379 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:09:02.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 6 -lt 21474836488 2024-06-03T05:09:02.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T05:09:03.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-03T05:09:03.792 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:09:04.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836489 -lt 21474836488 2024-06-03T05:09:04.194 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T05:09:04.195 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:09:04.195 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:09:04.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-03T05:09:04.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T05:09:04.899 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T05:09:04.899 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T05:09:04.899 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T05:09:04.899 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T05:09:04.899 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T05:09:04.899 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T05:09:05.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T05:09:05.345 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T05:09:05.345 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:09:05.345 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:09:05.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-03T05:09:05.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T05:09:05.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T05:09:05.927 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1380: test_get_num_active_clean: get_num_active_clean 2024-06-03T05:09:05.927 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T05:09:05.927 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T05:09:05.927 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T05:09:05.927 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T05:09:05.928 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T05:09:06.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1380: test_get_num_active_clean: local num_active_clean=4 2024-06-03T05:09:06.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1381: test_get_num_active_clean: test 4 = 4 2024-06-03T05:09:06.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1382: test_get_num_active_clean: teardown td/ceph-helpers 2024-06-03T05:09:06.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:09:06.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:09:06.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:09:06.356 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:09:06.356 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:09:06.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:09:06.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:09:06.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:09:06.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:09:06.463 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:09:06.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:09:06.464 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:09:06.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:09:06.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:09:06.465 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:09:06.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:09:06.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:09:06.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:09:06.467 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:09:06.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:09:06.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:09:06.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:09:06.474 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:09:06.475 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:09:06.475 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:09:06.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:09:06.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:09:06.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:09:06.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:09:06.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_get_num_active_or_peered td/ceph-helpers 2024-06-03T05:09:06.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1402: test_get_num_active_or_peered: local dir=td/ceph-helpers 2024-06-03T05:09:06.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1404: test_get_num_active_or_peered: setup td/ceph-helpers 2024-06-03T05:09:06.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-03T05:09:06.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-03T05:09:06.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:09:06.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:09:06.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:09:06.478 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:09:06.478 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:09:06.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:09:06.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:09:06.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:09:06.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:09:06.479 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:09:06.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:09:06.479 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:09:06.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:09:06.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:09:06.480 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:09:06.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:09:06.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:09:06.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:09:06.482 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:09:06.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:09:06.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:09:06.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:09:06.485 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:09:06.485 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:09:06.485 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:09:06.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:09:06.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:09:06.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:09:06.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-03T05:09:06.486 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:09:06.486 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:09:06.486 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:09:06.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70897 2024-06-03T05:09:06.488 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:09:06.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:09:06.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:09:06.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-03T05:09:06.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1405: test_get_num_active_or_peered: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:09:06.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-03T05:09:06.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:09:06.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:09:06.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:09:06.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-03T05:09:06.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:09:06.893 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:09:06.893 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:09:06.894 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:09:06.894 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:09:06.894 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:09:06.894 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:09:06.894 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:09:06.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:09:07.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:09:07.098 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:09:07.098 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:09:07.098 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:09:07.098 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:09:07.098 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:09:07.099 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:09:07.099 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:09:07.099 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:09:07.100 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:09:07.100 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:09:07.100 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:09:07.100 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:09:07.101 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:09:07.101 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get fsid 2024-06-03T05:09:07.212 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:09:07.212 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:09:07.212 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:09:07.212 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:09:07.213 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:09:07.213 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:09:07.213 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:09:07.213 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:09:07.213 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:09:07.213 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:09:07.213 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:09:07.213 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:09:07.214 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:09:07.214 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get mon_host 2024-06-03T05:09:07.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1406: test_get_num_active_or_peered: run_mgr td/ceph-helpers x 2024-06-03T05:09:07.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-03T05:09:07.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T05:09:07.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T05:09:07.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T05:09:07.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-03T05:09:07.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T05:09:08.218 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T05:09:08.218 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:09:08.218 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:09:08.219 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:09:08.219 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:09:08.219 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:09:08.219 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:09:08.221 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:09:08.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:09:08.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1407: test_get_num_active_or_peered: run_osd td/ceph-helpers 0 2024-06-03T05:09:08.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:09:08.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:09:08.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T05:09:08.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:09:08.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-03T05:09:08.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:09:08.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:09:08.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:09:08.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:09:08.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-03T05:09:08.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-03T05:09:08.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:09:08.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:09:08.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:09:08.257 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:09:08.257 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:09:08.257 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:09:08.257 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:09:08.257 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:09:08.257 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:09:08.257 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:09:08.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:09:08.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:09:08.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:09:08.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:09:08.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:09:08.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:09:08.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:09:08.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:09:08.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:09:08.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:09:08.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:09:08.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:09:08.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-03T05:09:08.259 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:09:08.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dcba0f31-40a6-4819-b70e-65aa0711efb4 2024-06-03T05:09:08.259 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 dcba0f31-40a6-4819-b70e-65aa0711efb4 2024-06-03T05:09:08.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 dcba0f31-40a6-4819-b70e-65aa0711efb4' 2024-06-03T05:09:08.260 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:09:08.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD0T11mgD5tEBAAMQWjVpxopXTTfuu3QXB5xA== 2024-06-03T05:09:08.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD0T11mgD5tEBAAMQWjVpxopXTTfuu3QXB5xA=="}' 2024-06-03T05:09:08.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dcba0f31-40a6-4819-b70e-65aa0711efb4 -i td/ceph-helpers/0/new.json 2024-06-03T05:09:08.584 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:09:08.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-03T05:09:08.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD0T11mgD5tEBAAMQWjVpxopXTTfuu3QXB5xA== --osd-uuid dcba0f31-40a6-4819-b70e-65aa0711efb4 2024-06-03T05:09:08.629 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:09:08.627+0000 7fd8bfe997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:09:08.671 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:09:08.667+0000 7fd8bfe997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:09:08.672 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:09:08.667+0000 7fd8bfe997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:09:08.672 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:09:08.671+0000 7fd8bfe997c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:09:08.672 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:09:08.671+0000 7fd8bfe997c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:09:08.673 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:09:08.671+0000 7fd8bfe997c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:09:08.710 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:09:08.707+0000 7fd8bfe997c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-03T05:09:14.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-03T05:09:14.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:09:14.971 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T05:09:14.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T05:09:14.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:09:15.520 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:09:15.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T05:09:15.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:09:15.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:09:15.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:09:15.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:09:15.551 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:09:15.547+0000 7fce2e55f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:09:15.554 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:09:15.551+0000 7fce2e55f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:09:15.555 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:09:15.551+0000 7fce2e55f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:09:15.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T05:09:15.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:09:15.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:09:15.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:09:15.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:09:15.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:09:15.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:09:15.925 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:09:15.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:09:15.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:09:16.130 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:09:16.127+0000 7fce2e55f7c0 -1 Falling back to public interface 2024-06-03T05:09:16.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:09:17.337 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:09:17.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:09:17.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:09:17.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:09:17.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:09:17.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:09:17.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:09:18.677 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:09:18.675+0000 7fce2e55f7c0 -1 osd.0 0 log_to_monitors true 2024-06-03T05:09:18.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:09:18.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:09:18.742 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:09:18.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:09:18.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:09:18.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:09:19.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:09:20.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:09:20.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:09:20.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:09:20.410 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:09:20.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:09:20.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:09:20.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:09:21.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:09:21.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:09:21.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:09:21.884 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:09:21.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:09:21.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:09:22.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:09:23.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:09:23.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:09:23.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:09:23.323 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:09:23.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:09:23.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:09:23.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:09:24.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:09:24.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:09:24.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:09:24.863 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:09:24.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:09:24.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:09:25.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:09:26.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:09:26.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:09:26.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:09:26.302 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:09:26.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:09:26.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:09:26.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:09:27.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:09:27.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:09:27.866 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:09:27.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:09:27.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:09:27.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:09:28.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:09:29.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:09:29.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:09:29.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:09:29.313 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:09:29.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:09:29.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:09:29.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:09:30.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:09:30.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:09:30.885 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:09:30.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:09:30.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:09:30.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:09:31.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:09:32.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:09:32.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:09:32.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:09:32.327 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:09:32.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:09:32.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:09:33.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:09:34.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:09:34.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:09:34.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:09:34.823 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:09:34.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:09:34.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:09:35.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:09:36.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:09:36.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:09:36.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:09:36.964 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:09:36.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:09:36.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:09:38.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:09:39.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:09:39.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:09:39.211 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:09:39.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:09:39.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:09:39.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:09:40.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:09:41.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:09:41.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:09:41.086 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:09:41.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:09:41.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:09:41.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:09:42.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:09:43.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:09:43.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:09:43.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:09:43.023 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:09:43.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:09:43.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:09:43.542 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/579366385,v1:127.0.0.1:6803/579366385] [v2:127.0.0.1:6804/579366385,v1:127.0.0.1:6805/579366385] exists,up dcba0f31-40a6-4819-b70e-65aa0711efb4 2024-06-03T05:09:43.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:09:43.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:09:43.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:09:43.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1408: test_get_num_active_or_peered: create_rbd_pool 2024-06-03T05:09:43.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-03T05:09:44.009 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' does not exist 2024-06-03T05:09:44.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-03T05:09:44.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-03T05:09:44.508 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' already exists 2024-06-03T05:09:44.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T05:09:45.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-03T05:09:46.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1409: test_get_num_active_or_peered: wait_for_clean 2024-06-03T05:09:46.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T05:09:46.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T05:09:46.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T05:09:46.676 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T05:09:46.677 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:09:46.677 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:09:46.677 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:09:46.677 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:09:46.677 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:09:46.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T05:09:46.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T05:09:46.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T05:09:46.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T05:09:46.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T05:09:46.798 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T05:09:47.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-06-03T05:09:47.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T05:09:47.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:09:47.218 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T05:09:47.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836489 2024-06-03T05:09:47.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836489 2024-06-03T05:09:47.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489' 2024-06-03T05:09:47.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:09:47.398 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836489 2024-06-03T05:09:47.398 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:09:47.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T05:09:47.401 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836489 2024-06-03T05:09:47.401 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:09:47.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836489 2024-06-03T05:09:47.402 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836489 2024-06-03T05:09:47.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2024-06-03T05:09:47.403 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:09:47.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836487 -lt 21474836489 2024-06-03T05:09:47.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T05:09:48.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T05:09:48.806 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:09:49.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836489 -lt 21474836489 2024-06-03T05:09:49.215 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T05:09:49.215 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:09:49.215 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:09:49.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-03T05:09:49.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T05:09:49.737 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T05:09:49.737 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T05:09:49.737 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T05:09:49.737 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T05:09:49.738 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T05:09:49.738 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T05:09:50.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T05:09:50.159 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T05:09:50.159 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:09:50.159 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:09:50.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-03T05:09:50.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T05:09:50.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T05:09:50.689 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1410: test_get_num_active_or_peered: get_num_active_or_peered 2024-06-03T05:09:50.689 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1394: get_num_active_or_peered: local expression 2024-06-03T05:09:50.689 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2024-06-03T05:09:50.689 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2024-06-03T05:09:50.689 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: get_num_active_or_peered: ceph --format json pg dump pgs 2024-06-03T05:09:50.689 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2024-06-03T05:09:51.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1410: test_get_num_active_or_peered: local num_peered=4 2024-06-03T05:09:51.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1411: test_get_num_active_or_peered: test 4 = 4 2024-06-03T05:09:51.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1412: test_get_num_active_or_peered: teardown td/ceph-helpers 2024-06-03T05:09:51.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:09:51.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:09:51.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:09:51.074 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:09:51.074 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:09:51.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:09:51.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:09:51.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:09:51.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:09:51.383 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:09:51.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:09:51.385 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:09:51.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:09:51.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:09:51.387 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:09:51.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:09:51.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:09:51.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:09:51.389 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:09:51.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:09:51.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:09:51.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:09:51.396 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:09:51.396 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:09:51.396 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:09:51.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:09:51.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:09:51.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:09:51.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:09:51.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_get_num_pgs td/ceph-helpers 2024-06-03T05:09:51.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1429: test_get_num_pgs: local dir=td/ceph-helpers 2024-06-03T05:09:51.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1431: test_get_num_pgs: setup td/ceph-helpers 2024-06-03T05:09:51.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-03T05:09:51.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-03T05:09:51.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:09:51.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:09:51.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:09:51.399 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:09:51.399 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:09:51.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:09:51.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:09:51.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:09:51.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:09:51.402 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:09:51.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:09:51.403 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:09:51.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:09:51.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:09:51.404 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:09:51.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:09:51.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:09:51.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:09:51.406 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:09:51.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:09:51.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:09:51.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:09:51.409 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:09:51.409 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:09:51.409 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:09:51.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:09:51.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:09:51.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:09:51.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-03T05:09:51.412 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:09:51.412 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:09:51.413 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:09:51.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70897 2024-06-03T05:09:51.415 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:09:51.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:09:51.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:09:51.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-03T05:09:51.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: test_get_num_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:09:51.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-03T05:09:51.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:09:51.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:09:51.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:09:51.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-03T05:09:51.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:09:51.914 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:09:51.914 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:09:51.914 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:09:51.914 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:09:51.915 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:09:51.915 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:09:51.915 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:09:51.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:09:52.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:09:52.118 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:09:52.118 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:09:52.118 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:09:52.118 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:09:52.120 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:09:52.120 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:09:52.120 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:09:52.120 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:09:52.120 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:09:52.120 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:09:52.120 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:09:52.120 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:09:52.120 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:09:52.120 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get fsid 2024-06-03T05:09:52.234 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:09:52.234 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:09:52.234 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:09:52.234 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:09:52.236 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:09:52.236 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:09:52.236 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:09:52.236 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:09:52.236 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:09:52.236 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:09:52.236 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:09:52.236 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:09:52.236 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:09:52.236 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get mon_host 2024-06-03T05:09:52.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: test_get_num_pgs: run_mgr td/ceph-helpers x 2024-06-03T05:09:52.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-03T05:09:52.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T05:09:52.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T05:09:52.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T05:09:52.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-03T05:09:52.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T05:09:53.239 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T05:09:53.239 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:09:53.239 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:09:53.239 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:09:53.240 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:09:53.240 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:09:53.240 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:09:53.240 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:09:53.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:09:53.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: test_get_num_pgs: run_osd td/ceph-helpers 0 2024-06-03T05:09:53.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:09:53.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:09:53.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T05:09:53.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:09:53.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-03T05:09:53.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:09:53.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:09:53.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:09:53.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:09:53.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-03T05:09:53.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-03T05:09:53.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:09:53.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:09:53.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:09:53.277 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:09:53.277 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:09:53.277 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:09:53.277 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:09:53.277 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:09:53.277 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:09:53.277 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:09:53.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:09:53.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:09:53.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:09:53.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:09:53.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:09:53.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:09:53.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:09:53.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:09:53.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:09:53.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:09:53.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:09:53.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:09:53.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-03T05:09:53.279 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:09:53.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7ef6a82e-746a-41bc-ba52-3ceb8bdcd90e 2024-06-03T05:09:53.279 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 7ef6a82e-746a-41bc-ba52-3ceb8bdcd90e 2024-06-03T05:09:53.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 7ef6a82e-746a-41bc-ba52-3ceb8bdcd90e' 2024-06-03T05:09:53.280 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:09:53.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAhUF1mcuuJERAAktAGbs/WlCB8CwOAbwQqmg== 2024-06-03T05:09:53.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAhUF1mcuuJERAAktAGbs/WlCB8CwOAbwQqmg=="}' 2024-06-03T05:09:53.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7ef6a82e-746a-41bc-ba52-3ceb8bdcd90e -i td/ceph-helpers/0/new.json 2024-06-03T05:09:53.571 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:09:53.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-03T05:09:53.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAhUF1mcuuJERAAktAGbs/WlCB8CwOAbwQqmg== --osd-uuid 7ef6a82e-746a-41bc-ba52-3ceb8bdcd90e 2024-06-03T05:09:53.626 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:09:53.623+0000 7f5eff58c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:09:53.648 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:09:53.643+0000 7f5eff58c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:09:53.649 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:09:53.647+0000 7f5eff58c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:09:53.650 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:09:53.647+0000 7f5eff58c7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:09:53.650 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:09:53.647+0000 7f5eff58c7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:09:53.650 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:09:53.647+0000 7f5eff58c7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:09:53.688 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:09:53.683+0000 7f5eff58c7c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-03T05:10:00.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-03T05:10:00.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:10:00.057 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T05:10:00.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T05:10:00.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:10:00.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T05:10:00.614 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:10:00.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:10:00.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:10:00.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:10:00.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:10:00.646 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:10:00.643+0000 7fbe4da507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:10:00.681 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:10:00.679+0000 7fbe4da507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:10:00.683 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:10:00.679+0000 7fbe4da507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:10:01.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T05:10:01.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:10:01.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:10:01.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:10:01.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:10:01.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:10:01.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:10:01.073 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:10:01.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:10:01.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:10:01.242 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:10:01.239+0000 7fbe4da507c0 -1 Falling back to public interface 2024-06-03T05:10:01.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:10:02.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:10:02.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:10:02.501 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:10:02.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:10:02.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:10:02.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:10:02.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:10:03.903 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:10:03.899+0000 7fbe4da507c0 -1 osd.0 0 log_to_monitors true 2024-06-03T05:10:03.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:10:03.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:10:03.913 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:10:03.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:10:03.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:10:03.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:10:04.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:10:05.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:10:05.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:10:05.598 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:10:05.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:10:05.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:10:05.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:10:06.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:10:07.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:10:07.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:10:07.036 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:10:07.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:10:07.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:10:07.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:10:07.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:10:08.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:10:08.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:10:08.525 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:10:08.526 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:10:08.526 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:10:08.526 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:10:09.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:10:10.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:10:10.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:10:10.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:10:10.006 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:10:10.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:10:10.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:10:10.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:10:11.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:10:11.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:10:11.450 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:10:11.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:10:11.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:10:11.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:10:11.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:10:12.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:10:12.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:10:12.929 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:10:12.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:10:12.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:10:12.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:10:13.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:10:14.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:10:14.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:10:14.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:10:14.411 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:10:14.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:10:14.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:10:14.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:10:15.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:10:15.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:10:15.864 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:10:15.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:10:15.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:10:15.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:10:16.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:10:17.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:10:17.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:10:17.303 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:10:17.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:10:17.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:10:17.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:10:17.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:10:18.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:10:18.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:10:18.797 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:10:18.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:10:18.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:10:18.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:10:19.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:10:20.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:10:20.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:10:20.833 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:10:20.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:10:20.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:10:20.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:10:21.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:10:22.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:10:22.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:10:22.896 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:10:22.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:10:22.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:10:22.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:10:23.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:10:24.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:10:24.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:10:24.804 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:10:24.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:10:24.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:10:24.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:10:25.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:10:26.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:10:26.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:10:26.589 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:10:26.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:10:26.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:10:26.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:10:27.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:10:28.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:10:28.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:10:28.693 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:10:28.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:10:28.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:10:28.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:10:29.113 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/427337195,v1:127.0.0.1:6803/427337195] [v2:127.0.0.1:6804/427337195,v1:127.0.0.1:6805/427337195] exists,up 7ef6a82e-746a-41bc-ba52-3ceb8bdcd90e 2024-06-03T05:10:29.113 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:10:29.113 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:10:29.113 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:10:29.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: test_get_num_pgs: create_rbd_pool 2024-06-03T05:10:29.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-03T05:10:29.634 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' does not exist 2024-06-03T05:10:29.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-03T05:10:29.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-03T05:10:30.156 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' already exists 2024-06-03T05:10:30.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T05:10:31.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-03T05:10:32.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1436: test_get_num_pgs: wait_for_clean 2024-06-03T05:10:32.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T05:10:32.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T05:10:32.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T05:10:32.316 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T05:10:32.317 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:10:32.317 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:10:32.318 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:10:32.318 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:10:32.318 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:10:32.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T05:10:32.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T05:10:32.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T05:10:32.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T05:10:32.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T05:10:32.439 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T05:10:32.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-06-03T05:10:32.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T05:10:32.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:10:32.862 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T05:10:33.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836485 2024-06-03T05:10:33.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836485 2024-06-03T05:10:33.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836485' 2024-06-03T05:10:33.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:10:33.014 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836485 2024-06-03T05:10:33.014 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:10:33.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T05:10:33.015 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836485 2024-06-03T05:10:33.015 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:10:33.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836485 2024-06-03T05:10:33.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2024-06-03T05:10:33.016 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836485 2024-06-03T05:10:33.016 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:10:33.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836483 -lt 21474836485 2024-06-03T05:10:33.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T05:10:34.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T05:10:34.417 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:10:34.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836485 -lt 21474836485 2024-06-03T05:10:34.834 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T05:10:34.834 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:10:34.834 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:10:35.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-03T05:10:35.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T05:10:35.377 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T05:10:35.377 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T05:10:35.377 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T05:10:35.377 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T05:10:35.377 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T05:10:35.377 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T05:10:35.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T05:10:35.790 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T05:10:35.791 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:10:35.791 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:10:36.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-03T05:10:36.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T05:10:36.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T05:10:36.321 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1437: test_get_num_pgs: get_num_pgs 2024-06-03T05:10:36.322 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:10:36.322 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:10:36.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1437: test_get_num_pgs: local num_pgs=4 2024-06-03T05:10:36.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1438: test_get_num_pgs: test 4 -gt 0 2024-06-03T05:10:36.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1439: test_get_num_pgs: teardown td/ceph-helpers 2024-06-03T05:10:36.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:10:36.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:10:36.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:10:36.865 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:10:36.865 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:10:36.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:10:36.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:10:36.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:10:37.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:10:37.174 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:10:37.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:10:37.175 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:10:37.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:10:37.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:10:37.176 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:10:37.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:10:37.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:10:37.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:10:37.178 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:10:37.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:10:37.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:10:37.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:10:37.188 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:10:37.188 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:10:37.188 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:10:37.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:10:37.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:10:37.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:10:37.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:10:37.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_get_op_scheduler td/ceph-helpers 2024-06-03T05:10:37.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: test_get_op_scheduler: local dir=td/ceph-helpers 2024-06-03T05:10:37.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: test_get_op_scheduler: setup td/ceph-helpers 2024-06-03T05:10:37.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-03T05:10:37.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-03T05:10:37.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:10:37.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:10:37.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:10:37.192 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:10:37.192 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:10:37.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:10:37.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:10:37.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:10:37.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:10:37.194 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:10:37.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:10:37.195 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:10:37.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:10:37.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:10:37.197 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:10:37.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:10:37.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:10:37.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:10:37.198 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:10:37.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:10:37.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:10:37.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:10:37.202 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:10:37.202 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:10:37.202 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:10:37.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:10:37.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:10:37.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:10:37.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-03T05:10:37.205 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:10:37.205 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:10:37.205 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:10:37.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70897 2024-06-03T05:10:37.207 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:10:37.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:10:37.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:10:37.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-03T05:10:37.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: test_get_op_scheduler: run_mon td/ceph-helpers a 2024-06-03T05:10:37.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-03T05:10:37.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:10:37.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:10:37.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:10:37.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-03T05:10:37.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-06-03T05:10:37.644 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:10:37.644 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:10:37.644 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:10:37.645 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:10:37.645 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:10:37.645 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:10:37.645 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:10:37.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-03T05:10:37.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:10:37.992 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:10:37.992 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:10:37.992 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:10:37.992 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:10:37.993 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:10:37.993 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:10:37.993 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:10:37.993 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:10:37.993 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:10:37.993 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:10:37.993 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:10:37.994 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:10:37.994 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:10:37.994 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get fsid 2024-06-03T05:10:38.105 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:10:38.105 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:10:38.105 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:10:38.105 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:10:38.106 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:10:38.106 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:10:38.106 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:10:38.106 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:10:38.107 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:10:38.107 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:10:38.107 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:10:38.107 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:10:38.107 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:10:38.107 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get mon_host 2024-06-03T05:10:38.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: test_get_op_scheduler: run_mgr td/ceph-helpers x 2024-06-03T05:10:38.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-03T05:10:38.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T05:10:38.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T05:10:38.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T05:10:38.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-03T05:10:38.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T05:10:39.102 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T05:10:39.103 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:10:39.103 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:10:39.103 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:10:39.103 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:10:39.103 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:10:39.103 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:10:39.104 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:10:39.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:10:39.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: test_get_op_scheduler: run_osd td/ceph-helpers 0 --osd_op_queue=wpq 2024-06-03T05:10:39.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:10:39.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:10:39.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T05:10:39.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:10:39.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-03T05:10:39.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:10:39.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:10:39.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:10:39.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:10:39.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-03T05:10:39.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-03T05:10:39.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:10:39.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:10:39.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:10:39.140 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:10:39.140 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:10:39.140 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:10:39.141 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:10:39.141 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:10:39.141 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:10:39.141 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:10:39.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:10:39.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:10:39.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:10:39.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:10:39.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:10:39.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:10:39.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:10:39.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:10:39.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:10:39.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:10:39.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:10:39.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=wpq 2024-06-03T05:10:39.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-03T05:10:39.142 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:10:39.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=150cf4c6-177a-4b42-ae41-385c4a52244e 2024-06-03T05:10:39.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 150cf4c6-177a-4b42-ae41-385c4a52244e' 2024-06-03T05:10:39.143 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 150cf4c6-177a-4b42-ae41-385c4a52244e 2024-06-03T05:10:39.143 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:10:39.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBPUF1mf0Z5CRAAjIoMjk8M0yAv1NBSFUHRmA== 2024-06-03T05:10:39.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBPUF1mf0Z5CRAAjIoMjk8M0yAv1NBSFUHRmA=="}' 2024-06-03T05:10:39.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 150cf4c6-177a-4b42-ae41-385c4a52244e -i td/ceph-helpers/0/new.json 2024-06-03T05:10:39.443 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:10:39.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-03T05:10:39.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=wpq --mkfs --key AQBPUF1mf0Z5CRAAjIoMjk8M0yAv1NBSFUHRmA== --osd-uuid 150cf4c6-177a-4b42-ae41-385c4a52244e 2024-06-03T05:10:39.494 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:10:39.491+0000 7f05de9667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:10:39.545 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:10:39.543+0000 7f05de9667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:10:39.546 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:10:39.543+0000 7f05de9667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:10:39.547 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:10:39.543+0000 7f05de9667c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:10:39.547 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:10:39.543+0000 7f05de9667c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:10:39.547 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:10:39.543+0000 7f05de9667c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:10:39.593 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:10:39.591+0000 7f05de9667c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-03T05:10:45.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-03T05:10:45.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:10:45.793 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T05:10:45.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T05:10:45.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:10:46.360 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:10:46.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T05:10:46.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:10:46.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=wpq 2024-06-03T05:10:46.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:10:46.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:10:46.392 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:10:46.387+0000 7f13690ac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:10:46.394 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:10:46.391+0000 7f13690ac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:10:46.395 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:10:46.391+0000 7f13690ac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:10:46.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T05:10:46.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:10:46.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:10:46.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:10:46.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:10:46.806 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:10:46.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:10:46.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:10:46.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:10:46.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:10:46.958 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:10:46.955+0000 7f13690ac7c0 -1 Falling back to public interface 2024-06-03T05:10:47.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:10:48.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:10:48.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:10:48.242 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:10:48.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:10:48.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:10:48.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:10:48.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:10:49.600 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:10:49.595+0000 7f13690ac7c0 -1 osd.0 0 log_to_monitors true 2024-06-03T05:10:49.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:10:49.662 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:10:49.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:10:49.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:10:49.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:10:49.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:10:50.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:10:51.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:10:51.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:10:51.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:10:51.147 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:10:51.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:10:51.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:10:51.590 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/511143729,v1:127.0.0.1:6803/511143729] [v2:127.0.0.1:6804/511143729,v1:127.0.0.1:6805/511143729] exists,up 150cf4c6-177a-4b42-ae41-385c4a52244e 2024-06-03T05:10:51.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:10:51.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:10:51.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:10:51.592 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: test_get_op_scheduler: get_op_scheduler 0 2024-06-03T05:10:51.592 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2254: get_op_scheduler: local id=0 2024-06-03T05:10:51.592 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: get_op_scheduler: get_config osd 0 osd_op_queue 2024-06-03T05:10:51.592 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=osd 2024-06-03T05:10:51.592 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=0 2024-06-03T05:10:51.592 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_op_queue 2024-06-03T05:10:51.592 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_op_queue 2024-06-03T05:10:51.592 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path osd.0 2024-06-03T05:10:51.592 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T05:10:51.592 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T05:10:51.593 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:10:51.593 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:10:51.593 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:10:51.594 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-osd.0.asok 2024-06-03T05:10:51.594 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:10:51.594 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-osd.0.asok config get osd_op_queue 2024-06-03T05:10:51.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: test_get_op_scheduler: test wpq = wpq 2024-06-03T05:10:51.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: test_get_op_scheduler: run_osd td/ceph-helpers 1 --osd_op_queue=mclock_scheduler 2024-06-03T05:10:51.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:10:51.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:10:51.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-03T05:10:51.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:10:51.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-06-03T05:10:51.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:10:51.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:10:51.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:10:51.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:10:51.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-06-03T05:10:51.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-06-03T05:10:51.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:10:51.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:10:51.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:10:51.708 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:10:51.708 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:10:51.708 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:10:51.708 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:10:51.708 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:10:51.708 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:10:51.709 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:10:51.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:10:51.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:10:51.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:10:51.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:10:51.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:10:51.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:10:51.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:10:51.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:10:51.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:10:51.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:10:51.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:10:51.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-06-03T05:10:51.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-06-03T05:10:51.710 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:10:51.711 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 fb6a9d34-61b2-4e97-a119-94cacb5ea02a 2024-06-03T05:10:51.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fb6a9d34-61b2-4e97-a119-94cacb5ea02a 2024-06-03T05:10:51.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 fb6a9d34-61b2-4e97-a119-94cacb5ea02a' 2024-06-03T05:10:51.712 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:10:51.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBbUF1mgCGJKxAA5SDFBQdPHjLT9022M0g0tw== 2024-06-03T05:10:51.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBbUF1mgCGJKxAA5SDFBQdPHjLT9022M0g0tw=="}' 2024-06-03T05:10:51.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fb6a9d34-61b2-4e97-a119-94cacb5ea02a -i td/ceph-helpers/1/new.json 2024-06-03T05:10:52.256 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:10:52.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-06-03T05:10:52.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQBbUF1mgCGJKxAA5SDFBQdPHjLT9022M0g0tw== --osd-uuid fb6a9d34-61b2-4e97-a119-94cacb5ea02a 2024-06-03T05:10:52.306 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:10:52.303+0000 7fbc6d0367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:10:52.328 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:10:52.323+0000 7fbc6d0367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:10:52.329 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:10:52.327+0000 7fbc6d0367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:10:52.330 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:10:52.327+0000 7fbc6d0367c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-03T05:10:52.330 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:10:52.327+0000 7fbc6d0367c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-03T05:10:52.330 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:10:52.327+0000 7fbc6d0367c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-03T05:10:52.375 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:10:52.371+0000 7fbc6d0367c0 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-06-03T05:10:58.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-06-03T05:10:58.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:10:58.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-03T05:10:58.980 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-06-03T05:10:58.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:10:59.550 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-06-03T05:10:59.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-03T05:10:59.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:10:59.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-06-03T05:10:59.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:10:59.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:10:59.583 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:10:59.579+0000 7f7ae4eec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:10:59.586 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:10:59.583+0000 7f7ae4eec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:10:59.587 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:10:59.583+0000 7f7ae4eec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:11:00.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-03T05:11:00.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:11:00.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T05:11:00.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:11:00.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:11:00.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:11:00.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:11:00.010 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:11:00.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:11:00.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:11:00.138 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:11:00.135+0000 7f7ae4eec7c0 -1 Falling back to public interface 2024-06-03T05:11:00.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:11:01.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:11:01.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:11:01.414 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:11:01.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:11:01.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:11:01.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:11:01.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:11:02.683 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:11:02.679+0000 7f7ae4eec7c0 -1 osd.1 0 log_to_monitors true 2024-06-03T05:11:02.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:11:02.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:11:02.817 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:11:02.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:11:02.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:11:02.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:11:03.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:11:04.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:11:04.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:11:04.456 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:11:04.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:11:04.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:11:04.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:11:05.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:11:06.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:11:06.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:11:06.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:11:06.006 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:11:06.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:11:06.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:11:06.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:11:07.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:11:07.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:11:07.493 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:11:07.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:11:07.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:11:07.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:11:07.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:11:08.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:11:08.971 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:11:08.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:11:08.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:11:08.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:11:08.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:11:09.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:11:10.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:11:10.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:11:10.580 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:11:10.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:11:10.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:11:10.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:11:11.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:11:12.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:11:12.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:11:12.043 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:11:12.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:11:12.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:11:12.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:11:12.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:11:13.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:11:13.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:11:13.654 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:11:13.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:11:13.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:11:13.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:11:14.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:11:15.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:11:15.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:11:15.084 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:11:15.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:11:15.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:11:15.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:11:15.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:11:16.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:11:16.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:11:16.645 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:11:16.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:11:16.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:11:16.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:11:17.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:11:18.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:11:18.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:11:18.444 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:11:18.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:11:18.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:11:18.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:11:18.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:11:19.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:11:19.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:11:19.886 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:11:19.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:11:19.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:11:19.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:11:21.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:11:22.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:11:22.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:11:22.041 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:11:22.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:11:22.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:11:22.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:11:22.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:11:23.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:11:23.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:11:23.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:11:23.898 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:11:23.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:11:23.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:11:24.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:11:25.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:11:25.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:11:25.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:11:25.776 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:11:25.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:11:25.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:11:26.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:11:27.903 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:11:27.903 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:11:27.903 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:11:27.904 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:11:27.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:11:27.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:11:28.362 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/187997622,v1:127.0.0.1:6811/187997622] [v2:127.0.0.1:6812/187997622,v1:127.0.0.1:6813/187997622] exists,up fb6a9d34-61b2-4e97-a119-94cacb5ea02a 2024-06-03T05:11:28.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:11:28.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:11:28.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:11:28.363 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: test_get_op_scheduler: get_op_scheduler 1 2024-06-03T05:11:28.363 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2254: get_op_scheduler: local id=1 2024-06-03T05:11:28.363 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: get_op_scheduler: get_config osd 1 osd_op_queue 2024-06-03T05:11:28.363 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=osd 2024-06-03T05:11:28.363 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=1 2024-06-03T05:11:28.363 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_op_queue 2024-06-03T05:11:28.364 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_op_queue 2024-06-03T05:11:28.365 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path osd.1 2024-06-03T05:11:28.365 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-03T05:11:28.365 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-03T05:11:28.365 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:11:28.365 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:11:28.365 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:11:28.365 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-osd.1.asok 2024-06-03T05:11:28.365 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:11:28.365 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-osd.1.asok config get osd_op_queue 2024-06-03T05:11:28.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: test_get_op_scheduler: test mclock_scheduler = mclock_scheduler 2024-06-03T05:11:28.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: test_get_op_scheduler: teardown td/ceph-helpers 2024-06-03T05:11:28.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:11:28.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:11:28.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:11:28.483 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:11:28.484 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:11:28.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:11:28.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:11:28.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:11:28.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:11:28.794 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:11:28.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:11:28.797 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:11:28.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:11:28.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:11:28.798 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:11:28.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:11:28.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:11:28.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:11:28.801 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:11:28.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:11:28.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:11:28.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:11:28.812 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:11:28.812 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:11:28.812 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:11:28.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:11:28.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:11:28.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:11:28.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:11:28.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_get_osd_id_used_by_pgs td/ceph-helpers 2024-06-03T05:11:28.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1459: test_get_osd_id_used_by_pgs: local dir=td/ceph-helpers 2024-06-03T05:11:28.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1461: test_get_osd_id_used_by_pgs: setup td/ceph-helpers 2024-06-03T05:11:28.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-03T05:11:28.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-03T05:11:28.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:11:28.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:11:28.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:11:28.815 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:11:28.815 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:11:28.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:11:28.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:11:28.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:11:28.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:11:28.817 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:11:28.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:11:28.818 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:11:28.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:11:28.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:11:28.820 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:11:28.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:11:28.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:11:28.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:11:28.821 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:11:28.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:11:28.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:11:28.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:11:28.825 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:11:28.825 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:11:28.825 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:11:28.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:11:28.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:11:28.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:11:28.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-03T05:11:28.828 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:11:28.828 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:11:28.828 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:11:28.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70897 2024-06-03T05:11:28.829 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:11:28.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:11:28.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:11:28.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-03T05:11:28.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1462: test_get_osd_id_used_by_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:11:28.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-03T05:11:28.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:11:28.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:11:28.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:11:28.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-03T05:11:28.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:11:29.236 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:11:29.236 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:11:29.236 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:11:29.236 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:11:29.236 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:11:29.237 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:11:29.237 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:11:29.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:11:29.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:11:29.496 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:11:29.496 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:11:29.496 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:11:29.496 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:11:29.497 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:11:29.497 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:11:29.497 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:11:29.497 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:11:29.498 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:11:29.498 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:11:29.498 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:11:29.498 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:11:29.498 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:11:29.498 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get fsid 2024-06-03T05:11:29.613 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:11:29.613 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:11:29.613 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:11:29.613 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:11:29.614 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:11:29.614 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:11:29.614 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:11:29.614 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:11:29.615 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:11:29.615 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:11:29.615 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:11:29.615 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:11:29.615 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:11:29.615 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get mon_host 2024-06-03T05:11:29.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1463: test_get_osd_id_used_by_pgs: run_mgr td/ceph-helpers x 2024-06-03T05:11:29.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-03T05:11:29.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T05:11:29.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T05:11:29.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T05:11:29.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-03T05:11:29.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T05:11:30.619 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T05:11:30.619 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:11:30.619 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:11:30.619 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:11:30.619 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:11:30.620 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:11:30.620 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:11:30.620 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:11:30.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:11:30.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1464: test_get_osd_id_used_by_pgs: run_osd td/ceph-helpers 0 2024-06-03T05:11:30.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:11:30.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:11:30.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T05:11:30.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:11:30.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-03T05:11:30.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:11:30.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:11:30.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:11:30.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:11:30.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-03T05:11:30.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-03T05:11:30.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:11:30.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:11:30.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:11:30.655 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:11:30.655 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:11:30.655 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:11:30.655 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:11:30.655 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:11:30.655 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:11:30.655 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:11:30.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:11:30.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:11:30.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:11:30.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:11:30.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:11:30.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:11:30.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:11:30.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:11:30.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:11:30.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:11:30.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:11:30.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:11:30.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-03T05:11:30.657 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:11:30.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d1317f9b-bfed-4bf9-9d86-47ff428a7536 2024-06-03T05:11:30.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d1317f9b-bfed-4bf9-9d86-47ff428a7536' 2024-06-03T05:11:30.658 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 d1317f9b-bfed-4bf9-9d86-47ff428a7536 2024-06-03T05:11:30.658 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:11:30.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCCUF1m1w0bKBAATDl6gsfib1fQqZRTE2+uXg== 2024-06-03T05:11:30.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCCUF1m1w0bKBAATDl6gsfib1fQqZRTE2+uXg=="}' 2024-06-03T05:11:30.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d1317f9b-bfed-4bf9-9d86-47ff428a7536 -i td/ceph-helpers/0/new.json 2024-06-03T05:11:30.985 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:11:30.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-03T05:11:31.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCCUF1m1w0bKBAATDl6gsfib1fQqZRTE2+uXg== --osd-uuid d1317f9b-bfed-4bf9-9d86-47ff428a7536 2024-06-03T05:11:31.028 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:11:31.022+0000 7f8392d8d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:11:31.061 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:11:31.058+0000 7f8392d8d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:11:31.063 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:11:31.058+0000 7f8392d8d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:11:31.063 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:11:31.058+0000 7f8392d8d7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:11:31.063 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:11:31.058+0000 7f8392d8d7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:11:31.063 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:11:31.058+0000 7f8392d8d7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:11:31.093 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:11:31.090+0000 7f8392d8d7c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-03T05:11:37.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-03T05:11:37.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:11:37.344 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T05:11:37.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T05:11:37.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:11:37.898 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:11:37.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T05:11:37.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:11:37.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:11:37.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:11:37.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:11:37.929 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:11:37.926+0000 7fe0b6e897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:11:37.951 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:11:37.946+0000 7fe0b6e897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:11:37.952 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:11:37.946+0000 7fe0b6e897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:11:38.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T05:11:38.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:11:38.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:11:38.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:11:38.319 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:11:38.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:11:38.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:11:38.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:11:38.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:11:38.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:11:38.498 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:11:38.494+0000 7fe0b6e897c0 -1 Falling back to public interface 2024-06-03T05:11:38.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:11:39.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:11:39.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:11:39.727 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:11:39.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:11:39.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:11:39.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:11:40.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:11:41.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:11:41.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:11:41.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:11:41.129 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:11:41.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:11:41.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:11:41.193 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:11:41.190+0000 7fe0b6e897c0 -1 osd.0 0 log_to_monitors true 2024-06-03T05:11:41.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:11:42.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:11:42.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:11:42.708 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:11:42.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:11:42.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:11:42.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:11:43.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:11:44.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:11:44.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:11:44.209 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:11:44.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:11:44.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:11:44.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:11:44.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:11:45.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:11:45.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:11:45.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:11:45.667 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:11:45.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:11:45.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:11:46.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:11:47.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:11:47.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:11:47.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:11:47.181 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:11:47.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:11:47.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:11:47.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:11:48.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:11:48.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:11:48.662 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:11:48.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:11:48.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:11:48.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:11:49.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:11:50.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:11:50.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:11:50.166 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:11:50.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:11:50.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:11:50.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:11:50.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:11:51.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:11:51.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:11:51.633 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:11:51.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:11:51.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:11:51.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:11:52.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:11:53.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:11:53.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:11:53.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:11:53.180 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:11:53.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:11:53.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:11:53.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:11:54.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:11:54.649 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:11:54.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:11:54.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:11:54.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:11:54.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:11:55.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:11:56.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:11:56.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:11:56.638 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:11:56.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:11:56.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:11:56.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:11:57.443 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:11:58.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:11:58.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:11:58.446 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:11:58.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:11:58.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:11:58.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:11:59.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:12:00.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:12:00.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:12:00.402 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:12:00.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:12:00.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:12:00.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:12:01.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:12:02.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:12:02.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:12:02.255 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:12:02.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:12:02.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:12:02.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:12:03.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:12:04.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:12:04.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:12:04.334 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:12:04.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:12:04.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:12:04.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:12:05.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:12:06.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:12:06.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:12:06.153 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:12:06.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:12:06.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:12:06.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:12:06.600 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1223594454,v1:127.0.0.1:6803/1223594454] [v2:127.0.0.1:6804/1223594454,v1:127.0.0.1:6805/1223594454] exists,up d1317f9b-bfed-4bf9-9d86-47ff428a7536 2024-06-03T05:12:06.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:12:06.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:12:06.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:12:06.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1465: test_get_osd_id_used_by_pgs: create_rbd_pool 2024-06-03T05:12:06.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-03T05:12:07.014 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' does not exist 2024-06-03T05:12:07.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-03T05:12:07.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-03T05:12:07.691 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' already exists 2024-06-03T05:12:07.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T05:12:08.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-03T05:12:09.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1466: test_get_osd_id_used_by_pgs: wait_for_clean 2024-06-03T05:12:09.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T05:12:09.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T05:12:09.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T05:12:09.941 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T05:12:09.942 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:12:09.942 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:12:09.942 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:12:09.943 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:12:09.943 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:12:10.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T05:12:10.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T05:12:10.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T05:12:10.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T05:12:10.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T05:12:10.061 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T05:12:10.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-06-03T05:12:10.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T05:12:10.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:12:10.485 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T05:12:10.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836491 2024-06-03T05:12:10.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836491 2024-06-03T05:12:10.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836491' 2024-06-03T05:12:10.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:12:10.647 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836491 2024-06-03T05:12:10.647 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:12:10.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T05:12:10.649 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836491 2024-06-03T05:12:10.649 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:12:10.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836491 2024-06-03T05:12:10.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2024-06-03T05:12:10.650 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836491 2024-06-03T05:12:10.650 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:12:11.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836489 -lt 21474836491 2024-06-03T05:12:11.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T05:12:12.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T05:12:12.044 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:12:12.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836491 -lt 21474836491 2024-06-03T05:12:12.451 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T05:12:12.452 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:12:12.452 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:12:13.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-03T05:12:13.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T05:12:13.044 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T05:12:13.044 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T05:12:13.044 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T05:12:13.044 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T05:12:13.044 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T05:12:13.044 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T05:12:13.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T05:12:13.505 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T05:12:13.506 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:12:13.506 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:12:14.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-03T05:12:14.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T05:12:14.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T05:12:14.025 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1467: test_get_osd_id_used_by_pgs: uniq 2024-06-03T05:12:14.025 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1467: test_get_osd_id_used_by_pgs: get_osd_id_used_by_pgs 2024-06-03T05:12:14.025 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: ceph --format json pg dump pgs 2024-06-03T05:12:14.025 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: jq '.pg_stats | .[] | .up[], .acting[]' 2024-06-03T05:12:14.025 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: sort 2024-06-03T05:12:14.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1467: test_get_osd_id_used_by_pgs: local osd_ids=0 2024-06-03T05:12:14.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: test_get_osd_id_used_by_pgs: test 0 = 0 2024-06-03T05:12:14.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: test_get_osd_id_used_by_pgs: teardown td/ceph-helpers 2024-06-03T05:12:14.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:12:14.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:12:14.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:12:14.475 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:12:14.475 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:12:14.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:12:14.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:12:14.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:12:14.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:12:14.582 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:12:14.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:12:14.583 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:12:14.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:12:14.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:12:14.585 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:12:14.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:12:14.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:12:14.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:12:14.586 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:12:14.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:12:14.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:12:14.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:12:14.594 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:12:14.594 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:12:14.595 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:12:14.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:12:14.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:12:14.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:12:14.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:12:14.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_get_osds td/ceph-helpers 2024-06-03T05:12:14.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: test_get_osds: local dir=td/ceph-helpers 2024-06-03T05:12:14.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: test_get_osds: setup td/ceph-helpers 2024-06-03T05:12:14.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-03T05:12:14.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-03T05:12:14.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:12:14.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:12:14.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:12:14.598 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:12:14.598 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:12:14.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:12:14.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:12:14.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:12:14.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:12:14.599 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:12:14.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:12:14.599 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:12:14.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:12:14.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:12:14.600 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:12:14.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:12:14.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:12:14.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:12:14.601 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:12:14.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:12:14.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:12:14.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:12:14.604 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:12:14.604 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:12:14.604 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:12:14.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:12:14.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:12:14.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:12:14.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-03T05:12:14.606 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:12:14.606 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:12:14.606 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:12:14.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70897 2024-06-03T05:12:14.608 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:12:14.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:12:14.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:12:14.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-03T05:12:14.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: test_get_osds: run_mon td/ceph-helpers a --osd_pool_default_size=2 2024-06-03T05:12:14.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-03T05:12:14.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:12:14.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:12:14.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:12:14.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-03T05:12:14.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2024-06-03T05:12:15.003 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:12:15.003 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:12:15.003 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:12:15.003 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:12:15.004 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:12:15.004 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:12:15.004 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:12:15.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-06-03T05:12:15.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:12:15.212 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:12:15.212 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:12:15.212 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:12:15.213 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:12:15.214 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:12:15.214 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:12:15.214 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:12:15.214 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:12:15.214 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:12:15.214 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:12:15.214 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:12:15.214 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:12:15.215 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:12:15.215 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get fsid 2024-06-03T05:12:15.328 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:12:15.328 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:12:15.328 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:12:15.328 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:12:15.329 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:12:15.329 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:12:15.330 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:12:15.330 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:12:15.330 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:12:15.330 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:12:15.330 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:12:15.330 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:12:15.330 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:12:15.330 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get mon_host 2024-06-03T05:12:15.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: test_get_osds: run_mgr td/ceph-helpers x 2024-06-03T05:12:15.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-03T05:12:15.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T05:12:15.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T05:12:15.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T05:12:15.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-03T05:12:15.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T05:12:16.379 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T05:12:16.380 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:12:16.380 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:12:16.380 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:12:16.380 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:12:16.381 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:12:16.381 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:12:16.382 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:12:16.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:12:16.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: test_get_osds: run_osd td/ceph-helpers 0 2024-06-03T05:12:16.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:12:16.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:12:16.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T05:12:16.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:12:16.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-03T05:12:16.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:12:16.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:12:16.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:12:16.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:12:16.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-03T05:12:16.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-03T05:12:16.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:12:16.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:12:16.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:12:16.448 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:12:16.448 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:12:16.448 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:12:16.448 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:12:16.448 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:12:16.448 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:12:16.449 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:12:16.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:12:16.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:12:16.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:12:16.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:12:16.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:12:16.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:12:16.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:12:16.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:12:16.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:12:16.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:12:16.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:12:16.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:12:16.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-03T05:12:16.450 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:12:16.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c9622195-46b4-427c-9f16-55c5442d4391 2024-06-03T05:12:16.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c9622195-46b4-427c-9f16-55c5442d4391' 2024-06-03T05:12:16.451 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 c9622195-46b4-427c-9f16-55c5442d4391 2024-06-03T05:12:16.451 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:12:16.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCwUF1mqOIrHBAAPanQaCnj8Sem4F49HjY6Yg== 2024-06-03T05:12:16.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCwUF1mqOIrHBAAPanQaCnj8Sem4F49HjY6Yg=="}' 2024-06-03T05:12:16.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c9622195-46b4-427c-9f16-55c5442d4391 -i td/ceph-helpers/0/new.json 2024-06-03T05:12:16.786 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:12:16.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-03T05:12:16.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCwUF1mqOIrHBAAPanQaCnj8Sem4F49HjY6Yg== --osd-uuid c9622195-46b4-427c-9f16-55c5442d4391 2024-06-03T05:12:16.841 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:12:16.838+0000 7f67796e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:12:16.870 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:12:16.866+0000 7f67796e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:12:16.871 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:12:16.866+0000 7f67796e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:12:16.872 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:12:16.866+0000 7f67796e77c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:12:16.872 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:12:16.866+0000 7f67796e77c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:12:16.872 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:12:16.866+0000 7f67796e77c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:12:16.918 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:12:16.914+0000 7f67796e77c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-03T05:12:23.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-03T05:12:23.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:12:23.339 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T05:12:23.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T05:12:23.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:12:23.878 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:12:23.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T05:12:23.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:12:23.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:12:23.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:12:23.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:12:23.909 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:12:23.906+0000 7f255e3577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:12:23.943 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:12:23.938+0000 7f255e3577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:12:23.944 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:12:23.942+0000 7f255e3577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:12:24.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T05:12:24.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:12:24.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:12:24.321 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:12:24.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:12:24.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:12:24.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:12:24.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:12:24.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:12:24.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:12:24.518 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:12:24.514+0000 7f255e3577c0 -1 Falling back to public interface 2024-06-03T05:12:24.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:12:25.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:12:25.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:12:25.744 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:12:25.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:12:25.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:12:25.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:12:26.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:12:27.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:12:27.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:12:27.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:12:27.156 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:12:27.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:12:27.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:12:27.171 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:12:27.166+0000 7f255e3577c0 -1 osd.0 0 log_to_monitors true 2024-06-03T05:12:27.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:12:28.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:12:28.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:12:28.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:12:28.709 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:12:28.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:12:28.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:12:29.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:12:30.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:12:30.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:12:30.171 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:12:30.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:12:30.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:12:30.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:12:30.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:12:31.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:12:31.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:12:31.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:12:31.613 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:12:31.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:12:31.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:12:32.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:12:33.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:12:33.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:12:33.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:12:33.178 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:12:33.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:12:33.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:12:33.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:12:34.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:12:34.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:12:34.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:12:34.617 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:12:34.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:12:34.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:12:35.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:12:36.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:12:36.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:12:36.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:12:36.183 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:12:36.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:12:36.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:12:36.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:12:37.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:12:37.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:12:37.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:12:37.630 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:12:37.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:12:37.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:12:38.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:12:39.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:12:39.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:12:39.262 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:12:39.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:12:39.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:12:39.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:12:39.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:12:40.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:12:40.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:12:40.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:12:40.721 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:12:40.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:12:40.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:12:41.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:12:42.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:12:42.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:12:42.654 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:12:42.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:12:42.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:12:42.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:12:43.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:12:44.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:12:44.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:12:44.588 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:12:44.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:12:44.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:12:44.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:12:45.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:12:46.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:12:46.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:12:46.264 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:12:46.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:12:46.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:12:46.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:12:47.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:12:48.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:12:48.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:12:48.128 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:12:48.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:12:48.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:12:48.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:12:48.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:12:49.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:12:49.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:12:49.876 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:12:49.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:12:49.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:12:49.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:12:50.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:12:51.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:12:51.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:12:51.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:12:51.649 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:12:51.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:12:51.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:12:52.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:12:53.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:12:53.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:12:53.179 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-06-03T05:12:53.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T05:12:53.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:12:53.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:12:53.581 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1378177808,v1:127.0.0.1:6803/1378177808] [v2:127.0.0.1:6804/1378177808,v1:127.0.0.1:6805/1378177808] exists,up c9622195-46b4-427c-9f16-55c5442d4391 2024-06-03T05:12:53.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:12:53.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:12:53.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:12:53.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1037: test_get_osds: run_osd td/ceph-helpers 1 2024-06-03T05:12:53.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:12:53.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:12:53.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-03T05:12:53.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:12:53.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-06-03T05:12:53.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:12:53.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:12:53.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:12:53.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:12:53.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-06-03T05:12:53.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-06-03T05:12:53.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:12:53.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:12:53.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:12:53.584 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:12:53.584 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:12:53.585 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:12:53.586 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:12:53.586 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:12:53.586 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:12:53.587 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:12:53.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:12:53.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:12:53.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:12:53.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:12:53.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:12:53.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:12:53.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:12:53.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:12:53.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:12:53.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:12:53.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:12:53.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:12:53.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-06-03T05:12:53.587 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:12:53.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b8415aa0-540c-4223-a94a-69e1933c0a90 2024-06-03T05:12:53.588 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 b8415aa0-540c-4223-a94a-69e1933c0a90 2024-06-03T05:12:53.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 b8415aa0-540c-4223-a94a-69e1933c0a90' 2024-06-03T05:12:53.589 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:12:53.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDVUF1mMiNSJBAAzLU9KylIc8TNh1tIi6f5yA== 2024-06-03T05:12:53.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDVUF1mMiNSJBAAzLU9KylIc8TNh1tIi6f5yA=="}' 2024-06-03T05:12:53.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b8415aa0-540c-4223-a94a-69e1933c0a90 -i td/ceph-helpers/1/new.json 2024-06-03T05:12:54.092 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:12:54.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-06-03T05:12:54.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDVUF1mMiNSJBAAzLU9KylIc8TNh1tIi6f5yA== --osd-uuid b8415aa0-540c-4223-a94a-69e1933c0a90 2024-06-03T05:12:54.141 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:12:54.138+0000 7fa8b1a397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:12:54.160 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:12:54.158+0000 7fa8b1a397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:12:54.161 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:12:54.158+0000 7fa8b1a397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:12:54.162 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:12:54.158+0000 7fa8b1a397c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-03T05:12:54.162 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:12:54.158+0000 7fa8b1a397c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-03T05:12:54.162 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:12:54.158+0000 7fa8b1a397c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-03T05:12:54.200 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:12:54.194+0000 7fa8b1a397c0 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-06-03T05:13:00.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-06-03T05:13:00.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:13:00.955 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-06-03T05:13:00.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-03T05:13:00.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:13:01.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-03T05:13:01.524 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-06-03T05:13:01.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:13:01.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:13:01.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:13:01.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:13:01.551 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:13:01.546+0000 7f1f73ec17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:13:01.582 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:13:01.578+0000 7f1f73ec17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:13:01.583 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:13:01.578+0000 7f1f73ec17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:13:01.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-03T05:13:01.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:13:01.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T05:13:01.996 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:13:01.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:13:01.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:13:01.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:13:01.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:13:01.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:13:01.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:13:02.166 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:13:02.162+0000 7f1f73ec17c0 -1 Falling back to public interface 2024-06-03T05:13:02.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:13:03.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:13:03.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:13:03.422 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:13:03.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:13:03.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:13:03.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:13:03.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:13:04.840 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:13:04.834+0000 7f1f73ec17c0 -1 osd.1 0 log_to_monitors true 2024-06-03T05:13:04.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:13:04.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:13:04.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:13:04.844 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:13:04.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:13:04.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:13:05.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:13:06.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:13:06.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:13:06.372 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:13:06.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:13:06.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:13:06.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:13:06.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:13:07.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:13:07.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:13:07.867 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:13:07.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:13:07.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:13:07.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:13:08.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:13:09.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:13:09.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:13:09.305 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:13:09.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:13:09.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:13:09.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:13:09.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:13:10.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:13:10.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:13:10.838 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:13:10.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:13:10.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:13:10.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:13:11.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:13:12.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:13:12.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:13:12.326 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:13:12.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:13:12.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:13:12.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:13:12.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:13:13.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:13:13.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:13:13.858 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:13:13.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:13:13.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:13:13.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:13:14.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:13:15.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:13:15.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:13:15.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:13:15.297 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:13:15.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:13:15.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:13:15.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:13:16.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:13:16.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:13:16.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:13:16.894 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:13:16.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:13:16.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:13:17.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:13:18.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:13:18.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:13:18.384 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:13:18.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:13:18.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:13:18.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:13:18.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:13:19.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:13:19.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:13:19.984 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:13:19.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:13:19.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:13:19.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:13:20.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:13:21.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:13:21.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:13:21.942 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:13:21.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:13:21.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:13:21.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:13:22.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:13:23.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:13:23.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:13:23.719 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:13:23.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:13:23.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:13:23.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:13:24.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:13:25.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:13:25.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:13:25.906 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:13:25.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:13:25.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:13:25.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:13:26.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:13:27.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:13:27.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:13:27.792 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:13:27.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:13:27.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:13:27.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:13:28.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:13:29.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:13:29.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:13:29.807 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:13:29.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:13:29.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:13:29.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:13:30.360 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1310014703,v1:127.0.0.1:6811/1310014703] [v2:127.0.0.1:6812/1310014703,v1:127.0.0.1:6813/1310014703] exists,up b8415aa0-540c-4223-a94a-69e1933c0a90 2024-06-03T05:13:30.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:13:30.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:13:30.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:13:30.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: test_get_osds: create_rbd_pool 2024-06-03T05:13:30.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-03T05:13:30.813 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' does not exist 2024-06-03T05:13:30.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-03T05:13:30.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-03T05:13:31.547 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' already exists 2024-06-03T05:13:31.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T05:13:32.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-03T05:13:34.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1039: test_get_osds: wait_for_clean 2024-06-03T05:13:34.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T05:13:34.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T05:13:34.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T05:13:34.454 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T05:13:34.455 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:13:34.455 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:13:34.456 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:13:34.456 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:13:34.456 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:13:34.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T05:13:34.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T05:13:34.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T05:13:34.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T05:13:34.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T05:13:34.574 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T05:13:35.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T05:13:35.001 INFO:tasks.workunit.client.0.smithi088.stderr:1' 2024-06-03T05:13:35.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T05:13:35.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:13:35.002 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T05:13:35.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836497 2024-06-03T05:13:35.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836497 2024-06-03T05:13:35.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497' 2024-06-03T05:13:35.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:13:35.184 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T05:13:35.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672969 2024-06-03T05:13:35.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672969 2024-06-03T05:13:35.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497 1-42949672969' 2024-06-03T05:13:35.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:13:35.345 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:13:35.345 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836497 2024-06-03T05:13:35.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T05:13:35.346 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836497 2024-06-03T05:13:35.346 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:13:35.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836497 2024-06-03T05:13:35.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2024-06-03T05:13:35.347 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836497 2024-06-03T05:13:35.347 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:13:35.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836495 -lt 21474836497 2024-06-03T05:13:35.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T05:13:36.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T05:13:36.813 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:13:37.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836497 2024-06-03T05:13:37.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:13:37.262 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672969 2024-06-03T05:13:37.262 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:13:37.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T05:13:37.264 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672969 2024-06-03T05:13:37.264 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:13:37.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672969 2024-06-03T05:13:37.266 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672969 2024-06-03T05:13:37.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-06-03T05:13:37.266 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T05:13:37.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672970 -lt 42949672969 2024-06-03T05:13:37.714 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T05:13:37.714 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:13:37.715 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:13:38.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-03T05:13:38.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T05:13:38.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T05:13:38.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T05:13:38.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T05:13:38.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T05:13:38.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T05:13:38.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T05:13:38.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T05:13:38.708 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T05:13:38.709 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:13:38.709 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:13:39.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-03T05:13:39.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T05:13:39.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T05:13:39.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1040: test_get_osds: create_rbd_pool 2024-06-03T05:13:39.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-03T05:13:39.783 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' does not exist 2024-06-03T05:13:39.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-03T05:13:39.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-03T05:13:40.453 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' already exists 2024-06-03T05:13:40.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T05:13:41.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-03T05:13:42.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1041: test_get_osds: get_osds rbd GROUP 2024-06-03T05:13:42.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=rbd 2024-06-03T05:13:42.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1041: test_get_osds: grep --quiet '^[0-1] [0-1]$' 2024-06-03T05:13:42.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=GROUP 2024-06-03T05:13:42.877 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map rbd GROUP 2024-06-03T05:13:42.877 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-03T05:13:43.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-06-03T05:13:43.466 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-06-03T05:13:43.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-06-03T05:13:43.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1042: test_get_osds: teardown td/ceph-helpers 2024-06-03T05:13:43.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:13:43.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:13:43.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:13:43.467 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:13:43.467 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:13:43.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:13:43.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:13:43.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:13:43.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:13:43.777 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:13:43.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:13:43.779 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:13:43.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:13:43.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:13:43.781 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:13:43.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:13:43.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:13:43.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:13:43.783 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:13:43.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:13:43.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:13:43.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:13:43.799 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:13:43.799 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:13:43.799 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:13:43.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:13:43.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:13:43.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:13:43.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:13:43.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_get_pg td/ceph-helpers 2024-06-03T05:13:43.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: test_get_pg: local dir=td/ceph-helpers 2024-06-03T05:13:43.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: test_get_pg: setup td/ceph-helpers 2024-06-03T05:13:43.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-03T05:13:43.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-03T05:13:43.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:13:43.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:13:43.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:13:43.803 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:13:43.803 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:13:43.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:13:43.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:13:43.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:13:43.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:13:43.805 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:13:43.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:13:43.806 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:13:43.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:13:43.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:13:43.808 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:13:43.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:13:43.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:13:43.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:13:43.810 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:13:43.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:13:43.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:13:43.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:13:43.814 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:13:43.814 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:13:43.814 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:13:43.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:13:43.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:13:43.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:13:43.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-03T05:13:43.816 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:13:43.816 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:13:43.816 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:13:43.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70897 2024-06-03T05:13:43.818 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:13:43.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:13:43.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:13:43.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-03T05:13:43.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1103: test_get_pg: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:13:43.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-03T05:13:43.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:13:43.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:13:43.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:13:43.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-03T05:13:43.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:13:44.239 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:13:44.239 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:13:44.239 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:13:44.240 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:13:44.240 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:13:44.240 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:13:44.241 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:13:44.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:13:44.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:13:44.458 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:13:44.459 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:13:44.459 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:13:44.459 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:13:44.459 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:13:44.459 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:13:44.459 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:13:44.459 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:13:44.460 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:13:44.460 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:13:44.460 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:13:44.460 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:13:44.460 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:13:44.461 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get fsid 2024-06-03T05:13:44.575 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:13:44.575 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:13:44.575 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:13:44.575 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:13:44.576 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:13:44.576 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:13:44.576 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:13:44.576 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:13:44.577 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:13:44.577 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:13:44.577 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:13:44.577 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:13:44.577 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:13:44.577 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get mon_host 2024-06-03T05:13:44.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1104: test_get_pg: run_mgr td/ceph-helpers x 2024-06-03T05:13:44.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-03T05:13:44.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T05:13:44.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T05:13:44.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T05:13:44.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-03T05:13:44.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T05:13:45.563 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T05:13:45.563 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:13:45.563 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:13:45.564 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:13:45.564 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:13:45.564 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:13:45.564 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:13:45.565 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:13:45.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:13:45.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: test_get_pg: run_osd td/ceph-helpers 0 2024-06-03T05:13:45.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:13:45.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:13:45.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T05:13:45.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:13:45.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-03T05:13:45.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:13:45.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:13:45.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:13:45.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:13:45.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-03T05:13:45.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-03T05:13:45.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:13:45.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:13:45.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:13:45.604 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:13:45.604 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:13:45.604 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:13:45.604 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:13:45.604 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:13:45.604 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:13:45.604 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:13:45.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:13:45.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:13:45.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:13:45.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:13:45.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:13:45.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:13:45.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:13:45.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:13:45.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:13:45.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:13:45.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:13:45.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:13:45.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-03T05:13:45.605 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:13:45.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d7a25f19-383a-4c50-83d3-9add6c699051 2024-06-03T05:13:45.606 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 d7a25f19-383a-4c50-83d3-9add6c699051 2024-06-03T05:13:45.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d7a25f19-383a-4c50-83d3-9add6c699051' 2024-06-03T05:13:45.606 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:13:45.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAJUV1mWPQGJRAAW5Fu03F4LtfvMkrV4wUy4Q== 2024-06-03T05:13:45.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAJUV1mWPQGJRAAW5Fu03F4LtfvMkrV4wUy4Q=="}' 2024-06-03T05:13:45.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d7a25f19-383a-4c50-83d3-9add6c699051 -i td/ceph-helpers/0/new.json 2024-06-03T05:13:45.915 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:13:45.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-03T05:13:45.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAJUV1mWPQGJRAAW5Fu03F4LtfvMkrV4wUy4Q== --osd-uuid d7a25f19-383a-4c50-83d3-9add6c699051 2024-06-03T05:13:45.960 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:13:45.954+0000 7fd79fcd47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:13:45.982 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:13:45.978+0000 7fd79fcd47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:13:45.983 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:13:45.978+0000 7fd79fcd47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:13:45.984 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:13:45.978+0000 7fd79fcd47c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:13:45.984 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:13:45.978+0000 7fd79fcd47c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:13:45.984 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:13:45.978+0000 7fd79fcd47c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:13:46.022 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:13:46.018+0000 7fd79fcd47c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-03T05:13:52.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-03T05:13:52.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:13:52.298 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T05:13:52.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T05:13:52.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:13:52.868 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:13:52.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T05:13:52.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:13:52.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:13:52.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:13:52.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:13:52.900 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:13:52.894+0000 7f2bd7d3e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:13:52.922 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:13:52.918+0000 7f2bd7d3e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:13:52.924 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:13:52.918+0000 7f2bd7d3e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:13:53.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T05:13:53.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:13:53.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:13:53.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:13:53.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:13:53.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:13:53.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:13:53.281 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:13:53.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:13:53.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:13:53.479 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:13:53.474+0000 7f2bd7d3e7c0 -1 Falling back to public interface 2024-06-03T05:13:53.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:13:54.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:13:54.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:13:54.692 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:13:54.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:13:54.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:13:54.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:13:55.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:13:56.048 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:13:56.042+0000 7f2bd7d3e7c0 -1 osd.0 0 log_to_monitors true 2024-06-03T05:13:56.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:13:56.097 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:13:56.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:13:56.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:13:56.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:13:56.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:13:56.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:13:57.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:13:57.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:13:57.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:13:57.715 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:13:57.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:13:57.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:13:58.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:13:59.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:13:59.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:13:59.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:13:59.198 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:13:59.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:13:59.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:13:59.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:14:00.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:14:00.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:14:00.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:14:00.636 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:14:00.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:14:00.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:14:01.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:14:02.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:14:02.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:14:02.151 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:14:02.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:14:02.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:14:02.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:14:02.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:14:03.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:14:03.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:14:03.628 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:14:03.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:14:03.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:14:03.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:14:04.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:14:05.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:14:05.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:14:05.193 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:14:05.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:14:05.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:14:05.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:14:05.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:14:06.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:14:06.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:14:06.616 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:14:06.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:14:06.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:14:06.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:14:07.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:14:08.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:14:08.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:14:08.179 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:14:08.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:14:08.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:14:08.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:14:08.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:14:09.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:14:09.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:14:09.628 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:14:09.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:14:09.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:14:09.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:14:10.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:14:11.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:14:11.496 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:14:11.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:14:11.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:14:11.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:14:11.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:14:12.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:14:13.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:14:13.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:14:13.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:14:13.361 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:14:13.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:14:13.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:14:14.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:14:15.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:14:15.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:14:15.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:14:15.241 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:14:15.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:14:15.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:14:16.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:14:17.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:14:17.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:14:17.174 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:14:17.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:14:17.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:14:17.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:14:17.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:14:18.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:14:18.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:14:18.975 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:14:18.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:14:18.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:14:18.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:14:19.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:14:20.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:14:20.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:14:20.713 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:14:20.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:14:20.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:14:20.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:14:21.162 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1190834855,v1:127.0.0.1:6803/1190834855] [v2:127.0.0.1:6804/1190834855,v1:127.0.0.1:6805/1190834855] exists,up d7a25f19-383a-4c50-83d3-9add6c699051 2024-06-03T05:14:21.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:14:21.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:14:21.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:14:21.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: test_get_pg: create_rbd_pool 2024-06-03T05:14:21.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-03T05:14:21.574 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' does not exist 2024-06-03T05:14:21.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-03T05:14:21.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-03T05:14:22.142 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' already exists 2024-06-03T05:14:22.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T05:14:23.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-03T05:14:24.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: test_get_pg: wait_for_clean 2024-06-03T05:14:24.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T05:14:24.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T05:14:24.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T05:14:24.489 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T05:14:24.490 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:14:24.490 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:14:24.490 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:14:24.490 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:14:24.490 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:14:24.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T05:14:24.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T05:14:24.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T05:14:24.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T05:14:24.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T05:14:24.614 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T05:14:25.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-06-03T05:14:25.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T05:14:25.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:14:25.077 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T05:14:25.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836485 2024-06-03T05:14:25.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836485 2024-06-03T05:14:25.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836485' 2024-06-03T05:14:25.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:14:25.236 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836485 2024-06-03T05:14:25.236 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:14:25.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T05:14:25.237 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836485 2024-06-03T05:14:25.237 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:14:25.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836485 2024-06-03T05:14:25.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2024-06-03T05:14:25.238 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836485 2024-06-03T05:14:25.238 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:14:25.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836483 -lt 21474836485 2024-06-03T05:14:25.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T05:14:26.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T05:14:26.634 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:14:27.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836485 -lt 21474836485 2024-06-03T05:14:27.079 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T05:14:27.079 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:14:27.080 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:14:27.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-03T05:14:27.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T05:14:27.640 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T05:14:27.640 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T05:14:27.640 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T05:14:27.640 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T05:14:27.640 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T05:14:27.641 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T05:14:28.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T05:14:28.053 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T05:14:28.053 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:14:28.053 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:14:28.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-03T05:14:28.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T05:14:28.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T05:14:28.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1108: test_get_pg: get_pg rbd GROUP 2024-06-03T05:14:28.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-06-03T05:14:28.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=GROUP 2024-06-03T05:14:28.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1108: test_get_pg: grep --quiet '^[0-9]\.[0-9a-f][0-9a-f]*$' 2024-06-03T05:14:28.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd GROUP 2024-06-03T05:14:28.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-03T05:14:29.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1109: test_get_pg: teardown td/ceph-helpers 2024-06-03T05:14:29.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:14:29.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:14:29.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:14:29.210 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:14:29.210 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:14:29.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:14:29.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:14:29.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:14:29.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:14:29.520 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:14:29.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:14:29.522 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:14:29.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:14:29.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:14:29.523 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:14:29.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:14:29.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:14:29.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:14:29.526 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:14:29.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:14:29.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:14:29.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:14:29.536 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:14:29.536 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:14:29.536 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:14:29.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:14:29.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:14:29.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:14:29.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:14:29.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_get_primary td/ceph-helpers 2024-06-03T05:14:29.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1204: test_get_primary: local dir=td/ceph-helpers 2024-06-03T05:14:29.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: test_get_primary: setup td/ceph-helpers 2024-06-03T05:14:29.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-03T05:14:29.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-03T05:14:29.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:14:29.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:14:29.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:14:29.540 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:14:29.540 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:14:29.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:14:29.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:14:29.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:14:29.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:14:29.542 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:14:29.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:14:29.543 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:14:29.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:14:29.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:14:29.545 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:14:29.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:14:29.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:14:29.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:14:29.547 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:14:29.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:14:29.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:14:29.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:14:29.550 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:14:29.550 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:14:29.550 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:14:29.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:14:29.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:14:29.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:14:29.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-03T05:14:29.553 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:14:29.553 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:14:29.553 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:14:29.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70897 2024-06-03T05:14:29.554 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:14:29.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:14:29.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:14:29.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-03T05:14:29.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: test_get_primary: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:14:29.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-03T05:14:29.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:14:29.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:14:29.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:14:29.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-03T05:14:29.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:14:30.028 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:14:30.028 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:14:30.028 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:14:30.029 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:14:30.029 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:14:30.029 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:14:30.029 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:14:30.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:14:30.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:14:30.250 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:14:30.250 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:14:30.250 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:14:30.250 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:14:30.250 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:14:30.251 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:14:30.251 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:14:30.251 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:14:30.251 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:14:30.251 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:14:30.252 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:14:30.252 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:14:30.252 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:14:30.252 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get fsid 2024-06-03T05:14:30.368 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:14:30.368 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:14:30.368 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:14:30.368 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:14:30.368 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:14:30.370 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:14:30.370 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:14:30.370 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:14:30.370 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:14:30.370 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:14:30.370 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:14:30.370 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:14:30.370 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:14:30.370 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get mon_host 2024-06-03T05:14:30.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: test_get_primary: local osd=0 2024-06-03T05:14:30.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1209: test_get_primary: run_mgr td/ceph-helpers x 2024-06-03T05:14:30.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-03T05:14:30.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T05:14:30.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T05:14:30.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T05:14:30.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-03T05:14:30.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T05:14:31.377 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T05:14:31.377 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:14:31.377 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:14:31.378 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:14:31.378 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:14:31.378 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:14:31.378 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:14:31.379 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:14:31.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:14:31.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: test_get_primary: run_osd td/ceph-helpers 0 2024-06-03T05:14:31.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:14:31.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:14:31.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T05:14:31.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:14:31.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-03T05:14:31.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:14:31.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:14:31.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:14:31.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:14:31.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-03T05:14:31.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-03T05:14:31.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:14:31.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:14:31.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:14:31.417 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:14:31.417 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:14:31.417 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:14:31.417 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:14:31.417 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:14:31.417 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:14:31.417 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:14:31.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:14:31.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:14:31.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:14:31.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:14:31.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:14:31.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:14:31.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:14:31.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:14:31.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:14:31.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:14:31.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:14:31.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:14:31.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-03T05:14:31.419 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:14:31.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f49d095b-aeeb-40df-a86a-c670a4f4a4b5 2024-06-03T05:14:31.419 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 f49d095b-aeeb-40df-a86a-c670a4f4a4b5 2024-06-03T05:14:31.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f49d095b-aeeb-40df-a86a-c670a4f4a4b5' 2024-06-03T05:14:31.419 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:14:31.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA3UV1mAujzGRAAgFlV1PReFsBfaWnbcClwUw== 2024-06-03T05:14:31.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA3UV1mAujzGRAAgFlV1PReFsBfaWnbcClwUw=="}' 2024-06-03T05:14:31.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f49d095b-aeeb-40df-a86a-c670a4f4a4b5 -i td/ceph-helpers/0/new.json 2024-06-03T05:14:31.736 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:14:31.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-03T05:14:31.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA3UV1mAujzGRAAgFlV1PReFsBfaWnbcClwUw== --osd-uuid f49d095b-aeeb-40df-a86a-c670a4f4a4b5 2024-06-03T05:14:31.789 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:14:31.786+0000 7f40b72bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:14:31.813 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:14:31.810+0000 7f40b72bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:14:31.814 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:14:31.810+0000 7f40b72bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:14:31.815 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:14:31.810+0000 7f40b72bf7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:14:31.815 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:14:31.810+0000 7f40b72bf7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:14:31.815 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:14:31.810+0000 7f40b72bf7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:14:31.853 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:14:31.850+0000 7f40b72bf7c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-03T05:14:38.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-03T05:14:38.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:14:38.177 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T05:14:38.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T05:14:38.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:14:38.720 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:14:38.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T05:14:38.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:14:38.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:14:38.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:14:38.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:14:38.753 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:14:38.750+0000 7f9efd3f57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:14:38.786 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:14:38.782+0000 7f9efd3f57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:14:38.787 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:14:38.782+0000 7f9efd3f57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:14:39.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T05:14:39.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:14:39.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:14:39.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:14:39.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:14:39.163 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:14:39.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:14:39.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:14:39.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:14:39.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:14:39.346 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:14:39.342+0000 7f9efd3f57c0 -1 Falling back to public interface 2024-06-03T05:14:39.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:14:40.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:14:40.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:14:40.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:14:40.584 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:14:40.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:14:40.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:14:40.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:14:41.978 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:14:41.974+0000 7f9efd3f57c0 -1 osd.0 0 log_to_monitors true 2024-06-03T05:14:42.000 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:14:42.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:14:42.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:14:42.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:14:42.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:14:42.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:14:42.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:14:43.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:14:43.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:14:43.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:14:43.468 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:14:43.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:14:43.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:14:43.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:14:44.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:14:44.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:14:44.977 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:14:44.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:14:44.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:14:44.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:14:45.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:14:46.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:14:46.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:14:46.449 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:14:46.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:14:46.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:14:46.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:14:46.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:14:47.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:14:47.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:14:47.973 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:14:47.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:14:47.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:14:47.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:14:48.462 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:14:49.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:14:49.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:14:49.465 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:14:49.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:14:49.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:14:49.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:14:49.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:14:50.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:14:50.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:14:50.968 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:14:50.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:14:50.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:14:50.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:14:51.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:14:52.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:14:52.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:14:52.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:14:52.414 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:14:52.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:14:52.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:14:52.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:14:53.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:14:53.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:14:53.994 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:14:53.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:14:53.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:14:53.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:14:54.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:14:55.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:14:55.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:14:55.414 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:14:55.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:14:55.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:14:55.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:14:56.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:14:57.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:14:57.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:14:57.451 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:14:57.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:14:57.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:14:57.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:14:58.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:14:59.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:14:59.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:14:59.252 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:14:59.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:14:59.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:14:59.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:15:00.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:15:01.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:15:01.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:15:01.308 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:15:01.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:15:01.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:15:01.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:15:02.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:15:03.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:15:03.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:15:03.213 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:15:03.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:15:03.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:15:03.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:15:03.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:15:04.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:15:04.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:15:04.987 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:15:04.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:15:04.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:15:04.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:15:05.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:15:06.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:15:06.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:15:06.790 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:15:06.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:15:06.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:15:06.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:15:07.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:15:08.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:15:08.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:15:08.265 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-06-03T05:15:08.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T05:15:08.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:15:08.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:15:08.692 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2211795592,v1:127.0.0.1:6803/2211795592] [v2:127.0.0.1:6804/2211795592,v1:127.0.0.1:6805/2211795592] exists,up f49d095b-aeeb-40df-a86a-c670a4f4a4b5 2024-06-03T05:15:08.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:15:08.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:15:08.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:15:08.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: test_get_primary: create_rbd_pool 2024-06-03T05:15:08.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-03T05:15:09.089 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' does not exist 2024-06-03T05:15:09.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-03T05:15:09.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-03T05:15:09.644 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' already exists 2024-06-03T05:15:09.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T05:15:10.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-03T05:15:11.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: test_get_primary: wait_for_clean 2024-06-03T05:15:11.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T05:15:11.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T05:15:11.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T05:15:11.886 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T05:15:11.887 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:15:11.887 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:15:11.888 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:15:11.888 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:15:11.888 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:15:12.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T05:15:12.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T05:15:12.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T05:15:12.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T05:15:12.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T05:15:12.009 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T05:15:12.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-06-03T05:15:12.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T05:15:12.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:15:12.445 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T05:15:12.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836486 2024-06-03T05:15:12.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836486 2024-06-03T05:15:12.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836486' 2024-06-03T05:15:12.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:15:12.602 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836486 2024-06-03T05:15:12.603 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:15:12.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T05:15:12.604 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836486 2024-06-03T05:15:12.604 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:15:12.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836486 2024-06-03T05:15:12.605 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836486 2024-06-03T05:15:12.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-06-03T05:15:12.605 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:15:13.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836483 -lt 21474836486 2024-06-03T05:15:13.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T05:15:14.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T05:15:14.008 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:15:14.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836486 -lt 21474836486 2024-06-03T05:15:14.408 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T05:15:14.408 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:15:14.408 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:15:14.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-03T05:15:14.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T05:15:14.988 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T05:15:14.988 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T05:15:14.988 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T05:15:14.988 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T05:15:14.988 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T05:15:14.989 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T05:15:15.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T05:15:15.430 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T05:15:15.430 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:15:15.430 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:15:16.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-03T05:15:16.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T05:15:16.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T05:15:16.065 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: test_get_primary: get_primary rbd GROUP 2024-06-03T05:15:16.065 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-06-03T05:15:16.065 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=GROUP 2024-06-03T05:15:16.066 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd GROUP 2024-06-03T05:15:16.066 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-03T05:15:16.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: test_get_primary: test 0 = 0 2024-06-03T05:15:16.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: test_get_primary: teardown td/ceph-helpers 2024-06-03T05:15:16.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:15:16.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:15:16.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:15:16.485 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:15:16.485 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:15:16.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:15:16.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:15:16.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:15:16.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:15:16.592 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:15:16.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:15:16.593 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:15:16.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:15:16.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:15:16.594 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:15:16.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:15:16.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:15:16.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:15:16.598 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:15:16.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:15:16.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:15:16.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:15:16.603 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:15:16.603 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:15:16.603 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:15:16.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:15:16.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:15:16.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:15:16.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:15:16.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_get_timeout_delays td/ceph-helpers 2024-06-03T05:15:16.604 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1616: test_get_timeout_delays: get_timeout_delays 1 2024-06-03T05:15:16.605 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:15:16.605 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:15:16.605 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:15:16.605 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:15:16.605 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:15:16.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1616: test_get_timeout_delays: test '1 ' = '1 ' 2024-06-03T05:15:16.615 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1617: test_get_timeout_delays: get_timeout_delays 5 2024-06-03T05:15:16.615 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:15:16.615 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:15:16.616 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:15:16.616 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:15:16.616 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:15:16.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1617: test_get_timeout_delays: test '1 2 2 ' = '1 2 2 ' 2024-06-03T05:15:16.634 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1618: test_get_timeout_delays: get_timeout_delays 6 2024-06-03T05:15:16.634 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:15:16.634 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:15:16.634 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:15:16.634 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:15:16.634 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:15:16.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1618: test_get_timeout_delays: test '1 2 3 ' = '1 2 3 ' 2024-06-03T05:15:16.653 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1619: test_get_timeout_delays: get_timeout_delays 7 2024-06-03T05:15:16.653 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:15:16.653 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:15:16.654 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:15:16.654 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:15:16.654 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:15:16.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1619: test_get_timeout_delays: test '1 2 4 ' = '1 2 4 ' 2024-06-03T05:15:16.678 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1620: test_get_timeout_delays: get_timeout_delays 8 2024-06-03T05:15:16.679 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:15:16.679 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:15:16.679 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:15:16.679 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:15:16.679 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:15:16.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1620: test_get_timeout_delays: test '1 2 4 1 ' = '1 2 4 1 ' 2024-06-03T05:15:16.708 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1621: test_get_timeout_delays: get_timeout_delays 1 .1 2024-06-03T05:15:16.709 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:15:16.709 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:15:16.709 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:15:16.709 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:15:16.709 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:15:16.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1621: test_get_timeout_delays: test '0.1 0.2 0.4 0.3 ' = '0.1 0.2 0.4 0.3 ' 2024-06-03T05:15:16.735 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1622: test_get_timeout_delays: get_timeout_delays 1.5 .1 2024-06-03T05:15:16.736 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:15:16.736 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:15:16.736 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:15:16.736 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:15:16.736 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:15:16.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1622: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 ' = '0.1 0.2 0.4 0.8 ' 2024-06-03T05:15:16.768 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: test_get_timeout_delays: get_timeout_delays 5 .1 2024-06-03T05:15:16.769 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:15:16.769 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:15:16.769 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:15:16.769 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:15:16.769 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:15:16.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 1.9 ' = '0.1 0.2 0.4 0.8 1.6 1.9 ' 2024-06-03T05:15:16.811 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1624: test_get_timeout_delays: get_timeout_delays 6 .1 2024-06-03T05:15:16.812 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:15:16.812 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:15:16.812 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:15:16.812 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:15:16.812 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:15:16.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1624: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 2.9 ' = '0.1 0.2 0.4 0.8 1.6 2.9 ' 2024-06-03T05:15:16.853 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1625: test_get_timeout_delays: get_timeout_delays 6.3 .1 2024-06-03T05:15:16.853 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:15:16.853 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:15:16.854 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:15:16.854 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:15:16.854 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:15:16.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1625: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 ' = '0.1 0.2 0.4 0.8 1.6 3.2 ' 2024-06-03T05:15:16.899 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1626: test_get_timeout_delays: get_timeout_delays 20 .1 2024-06-03T05:15:16.899 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:15:16.899 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:15:16.900 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:15:16.900 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:15:16.900 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:15:16.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1626: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 7.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 7.3 ' 2024-06-03T05:15:16.960 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: test_get_timeout_delays: get_timeout_delays 300 .1 0 2024-06-03T05:15:16.961 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:15:16.961 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:15:16.961 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:15:16.961 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:15:16.961 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:15:17.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 12.8 25.6 51.2 102.4 95.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 12.8 25.6 51.2 102.4 95.3 ' 2024-06-03T05:15:17.049 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1628: test_get_timeout_delays: get_timeout_delays 300 .1 10 2024-06-03T05:15:17.049 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:15:17.049 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:15:17.049 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:15:17.049 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:15:17.049 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:15:17.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1628: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 7.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 7.3 ' 2024-06-03T05:15:17.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:15:17.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_is_clean td/ceph-helpers 2024-06-03T05:15:17.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1561: test_is_clean: local dir=td/ceph-helpers 2024-06-03T05:15:17.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: test_is_clean: setup td/ceph-helpers 2024-06-03T05:15:17.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-03T05:15:17.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-03T05:15:17.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:15:17.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:15:17.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:15:17.332 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:15:17.332 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:15:17.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:15:17.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:15:17.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:15:17.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:15:17.334 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:15:17.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:15:17.335 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:15:17.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:15:17.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:15:17.337 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:15:17.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:15:17.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:15:17.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:15:17.339 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:15:17.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:15:17.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:15:17.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:15:17.342 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:15:17.342 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:15:17.342 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:15:17.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:15:17.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:15:17.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:15:17.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-03T05:15:17.344 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:15:17.345 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:15:17.345 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:15:17.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70897 2024-06-03T05:15:17.347 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:15:17.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:15:17.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:15:17.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-03T05:15:17.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: test_is_clean: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:15:17.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-03T05:15:17.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:15:17.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:15:17.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:15:17.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-03T05:15:17.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:15:17.774 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:15:17.774 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:15:17.774 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:15:17.775 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:15:17.775 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:15:17.775 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:15:17.775 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:15:17.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:15:18.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:15:18.018 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:15:18.018 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:15:18.018 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:15:18.018 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:15:18.019 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:15:18.019 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:15:18.019 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:15:18.019 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:15:18.020 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:15:18.020 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:15:18.020 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:15:18.020 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:15:18.020 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:15:18.020 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get fsid 2024-06-03T05:15:18.130 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:15:18.131 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:15:18.131 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:15:18.131 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:15:18.132 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:15:18.132 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:15:18.132 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:15:18.132 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:15:18.132 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:15:18.132 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:15:18.132 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:15:18.132 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:15:18.133 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:15:18.133 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get mon_host 2024-06-03T05:15:18.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1565: test_is_clean: run_mgr td/ceph-helpers x 2024-06-03T05:15:18.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-03T05:15:18.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T05:15:18.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T05:15:18.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T05:15:18.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-03T05:15:18.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T05:15:19.149 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T05:15:19.150 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:15:19.150 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:15:19.150 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:15:19.150 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:15:19.150 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:15:19.151 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:15:19.151 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:15:19.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:15:19.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: test_is_clean: run_osd td/ceph-helpers 0 2024-06-03T05:15:19.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:15:19.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:15:19.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T05:15:19.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:15:19.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-03T05:15:19.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:15:19.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:15:19.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:15:19.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:15:19.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-03T05:15:19.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-03T05:15:19.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:15:19.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:15:19.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:15:19.189 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:15:19.189 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:15:19.189 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:15:19.189 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:15:19.189 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:15:19.189 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:15:19.189 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:15:19.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:15:19.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:15:19.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:15:19.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:15:19.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:15:19.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:15:19.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:15:19.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:15:19.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:15:19.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:15:19.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:15:19.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:15:19.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-03T05:15:19.191 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:15:19.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a3917034-68f4-4498-9dbb-8422757389ce 2024-06-03T05:15:19.191 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 a3917034-68f4-4498-9dbb-8422757389ce 2024-06-03T05:15:19.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a3917034-68f4-4498-9dbb-8422757389ce' 2024-06-03T05:15:19.192 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:15:19.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBnUV1mofxMDBAABiCvkvPMYyAJL+qDJ2pZdA== 2024-06-03T05:15:19.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBnUV1mofxMDBAABiCvkvPMYyAJL+qDJ2pZdA=="}' 2024-06-03T05:15:19.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a3917034-68f4-4498-9dbb-8422757389ce -i td/ceph-helpers/0/new.json 2024-06-03T05:15:19.498 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:15:19.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-03T05:15:19.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBnUV1mofxMDBAABiCvkvPMYyAJL+qDJ2pZdA== --osd-uuid a3917034-68f4-4498-9dbb-8422757389ce 2024-06-03T05:15:19.555 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:15:19.550+0000 7fc1027377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:15:19.575 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:15:19.570+0000 7fc1027377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:15:19.576 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:15:19.574+0000 7fc1027377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:15:19.577 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:15:19.574+0000 7fc1027377c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:15:19.577 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:15:19.574+0000 7fc1027377c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:15:19.577 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:15:19.574+0000 7fc1027377c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:15:19.615 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:15:19.610+0000 7fc1027377c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-03T05:15:25.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-03T05:15:25.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:15:25.940 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T05:15:25.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T05:15:25.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:15:26.508 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:15:26.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T05:15:26.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:15:26.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:15:26.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:15:26.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:15:26.542 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:15:26.538+0000 7fa6a2c987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:15:26.548 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:15:26.546+0000 7fa6a2c987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:15:26.549 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:15:26.546+0000 7fa6a2c987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:15:26.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T05:15:26.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:15:26.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:15:26.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:15:26.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:15:26.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:15:26.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:15:26.972 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:15:26.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:15:26.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:15:27.114 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:15:27.110+0000 7fa6a2c987c0 -1 Falling back to public interface 2024-06-03T05:15:27.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:15:28.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:15:28.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:15:28.370 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:15:28.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:15:28.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:15:28.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:15:28.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:15:29.670 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:15:29.666+0000 7fa6a2c987c0 -1 osd.0 0 log_to_monitors true 2024-06-03T05:15:29.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:15:29.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:15:29.775 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:15:29.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:15:29.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:15:29.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:15:30.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:15:31.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:15:31.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:15:31.415 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:15:31.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:15:31.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:15:31.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:15:31.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:15:32.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:15:32.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:15:32.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:15:32.873 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:15:32.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:15:32.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:15:33.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:15:34.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:15:34.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:15:34.310 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:15:34.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:15:34.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:15:34.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:15:34.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:15:35.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:15:35.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:15:35.791 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:15:35.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:15:35.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:15:35.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:15:36.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:15:37.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:15:37.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:15:37.251 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:15:37.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:15:37.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:15:37.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:15:37.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:15:38.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:15:38.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:15:38.829 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:15:38.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:15:38.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:15:38.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:15:39.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:15:40.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:15:40.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:15:40.259 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:15:40.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:15:40.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:15:40.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:15:41.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:15:42.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:15:42.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:15:42.009 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:15:42.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:15:42.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:15:42.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:15:42.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:15:43.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:15:43.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:15:43.509 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:15:43.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:15:43.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:15:43.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:15:44.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:15:45.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:15:45.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:15:45.427 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:15:45.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:15:45.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:15:45.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:15:46.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:15:47.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:15:47.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:15:47.234 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:15:47.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:15:47.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:15:47.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:15:48.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:15:49.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:15:49.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:15:49.043 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:15:49.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:15:49.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:15:49.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:15:49.779 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:15:50.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:15:50.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:15:50.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:15:50.781 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:15:50.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:15:50.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:15:51.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:15:52.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:15:52.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:15:52.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:15:52.812 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:15:52.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:15:52.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:15:53.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:15:54.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:15:54.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:15:54.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:15:54.661 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:15:54.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:15:54.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:15:55.121 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2896216644,v1:127.0.0.1:6803/2896216644] [v2:127.0.0.1:6804/2896216644,v1:127.0.0.1:6805/2896216644] exists,up a3917034-68f4-4498-9dbb-8422757389ce 2024-06-03T05:15:55.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:15:55.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:15:55.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:15:55.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1567: test_is_clean: create_rbd_pool 2024-06-03T05:15:55.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-03T05:15:55.511 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' does not exist 2024-06-03T05:15:55.526 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-03T05:15:55.526 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-03T05:15:56.197 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' already exists 2024-06-03T05:15:56.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T05:15:57.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-03T05:15:58.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1568: test_is_clean: wait_for_clean 2024-06-03T05:15:58.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T05:15:58.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T05:15:58.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T05:15:58.448 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T05:15:58.448 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:15:58.449 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:15:58.449 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:15:58.449 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:15:58.449 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:15:58.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T05:15:58.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T05:15:58.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T05:15:58.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T05:15:58.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T05:15:58.571 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T05:15:58.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-06-03T05:15:58.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T05:15:58.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:15:58.991 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T05:15:59.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836490 2024-06-03T05:15:59.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836490 2024-06-03T05:15:59.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836490' 2024-06-03T05:15:59.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:15:59.159 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836490 2024-06-03T05:15:59.159 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:15:59.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T05:15:59.160 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836490 2024-06-03T05:15:59.160 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:15:59.161 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836490 2024-06-03T05:15:59.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836490 2024-06-03T05:15:59.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2024-06-03T05:15:59.162 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:15:59.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836488 -lt 21474836490 2024-06-03T05:15:59.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T05:16:00.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T05:16:00.550 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:16:00.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836490 -lt 21474836490 2024-06-03T05:16:00.979 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T05:16:00.980 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:16:00.980 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:16:01.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-03T05:16:01.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T05:16:01.540 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T05:16:01.540 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T05:16:01.540 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T05:16:01.541 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T05:16:01.541 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T05:16:01.541 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T05:16:01.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T05:16:01.982 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T05:16:01.983 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:16:01.983 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:16:02.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-03T05:16:02.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T05:16:02.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T05:16:02.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1569: test_is_clean: is_clean 2024-06-03T05:16:02.523 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: is_clean: get_num_pgs 2024-06-03T05:16:02.524 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:16:02.524 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:16:03.127 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: is_clean: num_pgs=4 2024-06-03T05:16:03.127 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1556: is_clean: test 4 '!=' 0 2024-06-03T05:16:03.127 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1557: is_clean: get_num_active_clean 2024-06-03T05:16:03.127 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T05:16:03.127 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T05:16:03.127 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T05:16:03.127 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T05:16:03.127 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T05:16:03.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1557: is_clean: test 4 = 4 2024-06-03T05:16:03.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1570: test_is_clean: teardown td/ceph-helpers 2024-06-03T05:16:03.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:16:03.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:16:03.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:16:03.515 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:16:03.515 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:16:03.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:16:03.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:16:03.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:16:03.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:16:03.622 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:16:03.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:16:03.623 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:16:03.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:16:03.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:16:03.625 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:16:03.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:16:03.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:16:03.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:16:03.627 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:16:03.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:16:03.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:16:03.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:16:03.635 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:16:03.636 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:16:03.636 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:16:03.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:16:03.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:16:03.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:16:03.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:16:03.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_kill_daemon td/ceph-helpers 2024-06-03T05:16:03.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:275: test_kill_daemon: local dir=td/ceph-helpers 2024-06-03T05:16:03.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:276: test_kill_daemon: setup td/ceph-helpers 2024-06-03T05:16:03.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-03T05:16:03.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-03T05:16:03.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:16:03.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:16:03.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:16:03.642 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:16:03.642 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:16:03.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:16:03.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:16:03.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:16:03.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:16:03.643 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:16:03.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:16:03.647 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:16:03.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:16:03.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:16:03.648 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:16:03.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:16:03.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:16:03.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:16:03.648 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:16:03.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:16:03.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:16:03.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:16:03.649 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:16:03.649 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:16:03.649 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:16:03.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:16:03.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:16:03.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:16:03.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-03T05:16:03.654 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:16:03.654 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:16:03.654 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:16:03.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70897 2024-06-03T05:16:03.654 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:16:03.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:16:03.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:16:03.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-03T05:16:03.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:277: test_kill_daemon: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:16:03.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-03T05:16:03.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:16:03.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:16:03.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:16:03.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-03T05:16:03.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:16:04.045 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:16:04.045 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:16:04.045 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:16:04.045 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:16:04.046 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:16:04.046 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:16:04.046 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:16:04.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:16:04.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:16:04.274 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:16:04.274 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:16:04.274 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:16:04.274 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:16:04.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:16:04.275 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:16:04.275 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:16:04.275 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:16:04.275 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:16:04.275 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:16:04.276 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:16:04.276 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:16:04.276 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:16:04.276 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get fsid 2024-06-03T05:16:04.385 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:16:04.386 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:16:04.386 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:16:04.386 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:16:04.388 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:16:04.388 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:16:04.389 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:16:04.389 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:16:04.389 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:16:04.389 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:16:04.389 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:16:04.389 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:16:04.390 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:16:04.390 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get mon_host 2024-06-03T05:16:04.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:278: test_kill_daemon: run_mgr td/ceph-helpers x 2024-06-03T05:16:04.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-03T05:16:04.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T05:16:04.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T05:16:04.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T05:16:04.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-03T05:16:04.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T05:16:05.410 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T05:16:05.410 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:16:05.410 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:16:05.410 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:16:05.411 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:16:05.411 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:16:05.411 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:16:05.411 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:16:05.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:16:05.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:279: test_kill_daemon: run_osd td/ceph-helpers 0 2024-06-03T05:16:05.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:16:05.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:16:05.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T05:16:05.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:16:05.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-03T05:16:05.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:16:05.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:16:05.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:16:05.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:16:05.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-03T05:16:05.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-03T05:16:05.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:16:05.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:16:05.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:16:05.452 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:16:05.452 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:16:05.453 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:16:05.453 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:16:05.453 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:16:05.453 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:16:05.453 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:16:05.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:16:05.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:16:05.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:16:05.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:16:05.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:16:05.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:16:05.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:16:05.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:16:05.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:16:05.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:16:05.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:16:05.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:16:05.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-03T05:16:05.454 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:16:05.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=71bbd79e-e06e-4eb4-8e1e-e8a04d07318a 2024-06-03T05:16:05.455 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 71bbd79e-e06e-4eb4-8e1e-e8a04d07318a 2024-06-03T05:16:05.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 71bbd79e-e06e-4eb4-8e1e-e8a04d07318a' 2024-06-03T05:16:05.455 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:16:05.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCVUV1mm3lhHBAAf8SrUM9XJebI1ycR4wyloA== 2024-06-03T05:16:05.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCVUV1mm3lhHBAAf8SrUM9XJebI1ycR4wyloA=="}' 2024-06-03T05:16:05.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 71bbd79e-e06e-4eb4-8e1e-e8a04d07318a -i td/ceph-helpers/0/new.json 2024-06-03T05:16:05.785 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:16:05.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-03T05:16:05.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCVUV1mm3lhHBAAf8SrUM9XJebI1ycR4wyloA== --osd-uuid 71bbd79e-e06e-4eb4-8e1e-e8a04d07318a 2024-06-03T05:16:05.830 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:16:05.826+0000 7effc659d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:16:05.854 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:16:05.850+0000 7effc659d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:16:05.856 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:16:05.850+0000 7effc659d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:16:05.857 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:16:05.854+0000 7effc659d7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:16:05.857 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:16:05.854+0000 7effc659d7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:16:05.857 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:16:05.854+0000 7effc659d7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:16:05.893 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:16:05.890+0000 7effc659d7c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-03T05:16:12.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-03T05:16:12.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:16:12.301 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T05:16:12.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T05:16:12.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:16:12.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T05:16:12.843 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:16:12.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:16:12.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:16:12.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:16:12.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:16:12.874 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:16:12.870+0000 7f6b0af9e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:16:12.876 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:16:12.874+0000 7f6b0af9e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:16:12.878 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:16:12.874+0000 7f6b0af9e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:16:13.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T05:16:13.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:16:13.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:16:13.300 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:16:13.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:16:13.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:16:13.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:16:13.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:16:13.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:16:13.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:16:13.425 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:16:13.422+0000 7f6b0af9e7c0 -1 Falling back to public interface 2024-06-03T05:16:13.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:16:14.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:16:14.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:16:14.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:16:14.725 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:16:14.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:16:14.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:16:15.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:16:16.094 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:16:16.090+0000 7f6b0af9e7c0 -1 osd.0 0 log_to_monitors true 2024-06-03T05:16:16.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:16:16.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:16:16.147 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:16:16.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:16:16.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:16:16.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:16:16.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:16:17.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:16:17.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:16:17.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:16:17.810 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:16:17.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:16:17.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:16:18.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:16:19.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:16:19.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:16:19.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:16:19.279 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:16:19.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:16:19.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:16:19.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:16:20.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:16:20.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:16:20.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:16:20.739 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:16:20.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:16:20.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:16:21.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:16:22.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:16:22.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:16:22.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:16:22.278 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:16:22.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:16:22.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:16:22.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:16:23.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:16:23.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:16:23.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:16:23.754 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:16:23.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:16:23.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:16:24.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:16:25.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:16:25.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:16:25.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:16:25.239 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:16:25.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:16:25.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:16:25.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:16:26.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:16:26.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:16:26.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:16:26.689 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:16:26.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:16:26.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:16:27.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:16:28.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:16:28.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:16:28.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:16:28.144 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:16:28.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:16:28.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:16:28.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:16:29.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:16:29.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:16:29.603 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:16:29.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:16:29.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:16:29.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:16:30.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:16:31.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:16:31.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:16:31.935 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:16:31.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:16:31.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:16:31.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:16:32.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:16:33.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:16:33.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:16:33.687 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:16:33.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:16:33.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:16:33.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:16:34.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:16:35.462 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:16:35.462 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:16:35.462 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:16:35.462 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:16:35.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:16:35.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:16:36.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:16:37.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:16:37.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:16:37.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:16:37.146 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:16:37.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:16:37.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:16:38.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:16:39.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:16:39.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:16:39.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:16:39.097 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:16:39.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:16:39.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:16:40.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:16:41.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:16:41.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:16:41.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:16:41.050 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:16:41.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:16:41.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:16:41.483 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3630845302,v1:127.0.0.1:6803/3630845302] [v2:127.0.0.1:6804/3630845302,v1:127.0.0.1:6805/3630845302] exists,up 71bbd79e-e06e-4eb4-8e1e-e8a04d07318a 2024-06-03T05:16:41.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:16:41.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:16:41.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:16:41.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:281: test_kill_daemon: name_prefix=osd 2024-06-03T05:16:41.485 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:282: test_kill_daemon: grep 'osd[^/]*\.pid' 2024-06-03T05:16:41.485 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:282: test_kill_daemon: find td/ceph-helpers 2024-06-03T05:16:41.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:282: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2024-06-03T05:16:41.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:289: test_kill_daemon: kill_daemon td/ceph-helpers/osd.0.pid 0 1 2024-06-03T05:16:41.486 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/osd.0.pid 2024-06-03T05:16:41.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=117389 2024-06-03T05:16:41.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=0 2024-06-03T05:16:41.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local delays=1 2024-06-03T05:16:41.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2024-06-03T05:16:41.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-06-03T05:16:41.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 117389 2024-06-03T05:16:41.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-06-03T05:16:41.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-06-03T05:16:41.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 1 2024-06-03T05:16:42.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-06-03T05:16:42.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 117389 2024-06-03T05:16:42.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-06-03T05:16:42.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-06-03T05:16:42.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0 2024-06-03T05:16:42.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 1 2024-06-03T05:16:42.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:293: test_kill_daemon: kill_daemon td/ceph-helpers/osd.0.pid TERM 2024-06-03T05:16:42.492 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/osd.0.pid 2024-06-03T05:16:42.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=117389 2024-06-03T05:16:42.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2024-06-03T05:16:42.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2024-06-03T05:16:42.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2024-06-03T05:16:42.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-06-03T05:16:42.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 117389 2024-06-03T05:16:42.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-06-03T05:16:42.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-06-03T05:16:42.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2024-06-03T05:16:42.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-06-03T05:16:42.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 117389 2024-06-03T05:16:42.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-06-03T05:16:42.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-06-03T05:16:42.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.2 2024-06-03T05:16:42.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-06-03T05:16:42.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 117389 2024-06-03T05:16:42.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2024-06-03T05:16:42.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2024-06-03T05:16:42.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2024-06-03T05:16:42.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:296: test_kill_daemon: name_prefix=mgr 2024-06-03T05:16:42.799 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:297: test_kill_daemon: find td/ceph-helpers 2024-06-03T05:16:42.800 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:297: test_kill_daemon: grep 'mgr[^/]*\.pid' 2024-06-03T05:16:42.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:297: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2024-06-03T05:16:42.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:301: test_kill_daemon: kill_daemon td/ceph-helpers/mgr.x.pid TERM 2024-06-03T05:16:42.802 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/mgr.x.pid 2024-06-03T05:16:42.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=116446 2024-06-03T05:16:42.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2024-06-03T05:16:42.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2024-06-03T05:16:42.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2024-06-03T05:16:42.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-06-03T05:16:42.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 116446 2024-06-03T05:16:42.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-06-03T05:16:42.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-06-03T05:16:42.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2024-06-03T05:16:42.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-06-03T05:16:42.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 116446 2024-06-03T05:16:42.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2024-06-03T05:16:42.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2024-06-03T05:16:42.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2024-06-03T05:16:42.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:304: test_kill_daemon: name_prefix=mon 2024-06-03T05:16:42.907 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:305: test_kill_daemon: find td/ceph-helpers 2024-06-03T05:16:42.907 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:305: test_kill_daemon: grep 'mon[^/]*\.pid' 2024-06-03T05:16:42.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:305: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2024-06-03T05:16:42.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:309: test_kill_daemon: kill_daemon td/ceph-helpers/mon.a.pid TERM 2024-06-03T05:16:42.910 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/mon.a.pid 2024-06-03T05:16:42.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=116361 2024-06-03T05:16:42.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2024-06-03T05:16:42.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2024-06-03T05:16:42.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2024-06-03T05:16:42.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-06-03T05:16:42.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 116361 2024-06-03T05:16:42.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-06-03T05:16:42.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-06-03T05:16:42.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2024-06-03T05:16:43.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-06-03T05:16:43.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 116361 2024-06-03T05:16:43.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2024-06-03T05:16:43.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2024-06-03T05:16:43.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2024-06-03T05:16:43.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:310: test_kill_daemon: timeout 5 ceph status 2024-06-03T05:16:48.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:313: test_kill_daemon: teardown td/ceph-helpers 2024-06-03T05:16:48.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:16:48.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:16:48.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:16:48.019 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:16:48.019 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:16:48.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:16:48.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:16:48.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:16:48.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:16:48.026 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:16:48.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:16:48.027 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:16:48.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:16:48.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:16:48.029 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:16:48.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:16:48.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:16:48.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:16:48.031 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:16:48.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:16:48.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:16:48.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:16:48.054 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:16:48.054 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:16:48.054 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:16:48.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:16:48.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:16:48.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:16:48.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:16:48.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_kill_daemons td/ceph-helpers 2024-06-03T05:16:48.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:366: test_kill_daemons: local dir=td/ceph-helpers 2024-06-03T05:16:48.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:367: test_kill_daemons: setup td/ceph-helpers 2024-06-03T05:16:48.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-03T05:16:48.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-03T05:16:48.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:16:48.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:16:48.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:16:48.057 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:16:48.057 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:16:48.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:16:48.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:16:48.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:16:48.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:16:48.059 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:16:48.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:16:48.060 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:16:48.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:16:48.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:16:48.062 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:16:48.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:16:48.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:16:48.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:16:48.064 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:16:48.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:16:48.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:16:48.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:16:48.067 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:16:48.067 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:16:48.067 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:16:48.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:16:48.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:16:48.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:16:48.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-03T05:16:48.070 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:16:48.070 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:16:48.070 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:16:48.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70897 2024-06-03T05:16:48.073 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:16:48.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:16:48.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:16:48.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-03T05:16:48.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:368: test_kill_daemons: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:16:48.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-03T05:16:48.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:16:48.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:16:48.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:16:48.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-03T05:16:48.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:16:48.459 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:16:48.460 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:16:48.460 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:16:48.460 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:16:48.460 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:16:48.460 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:16:48.461 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:16:48.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:16:48.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:16:48.682 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:16:48.682 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:16:48.682 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:16:48.682 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:16:48.682 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:16:48.682 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:16:48.682 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:16:48.683 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:16:48.683 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:16:48.683 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:16:48.683 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:16:48.683 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:16:48.684 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:16:48.684 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get fsid 2024-06-03T05:16:48.800 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:16:48.800 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:16:48.800 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:16:48.800 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:16:48.801 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:16:48.801 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:16:48.801 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:16:48.801 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:16:48.801 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:16:48.801 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:16:48.801 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:16:48.802 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:16:48.802 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:16:48.802 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get mon_host 2024-06-03T05:16:48.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:369: test_kill_daemons: run_mgr td/ceph-helpers x 2024-06-03T05:16:48.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-03T05:16:48.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T05:16:48.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T05:16:48.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T05:16:48.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-03T05:16:48.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T05:16:49.818 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T05:16:49.819 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:16:49.819 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:16:49.819 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:16:49.819 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:16:49.820 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:16:49.820 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:16:49.821 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:16:49.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:16:49.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:370: test_kill_daemons: run_osd td/ceph-helpers 0 2024-06-03T05:16:49.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:16:49.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:16:49.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T05:16:49.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:16:49.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-03T05:16:49.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:16:49.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:16:49.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:16:49.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:16:49.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-03T05:16:49.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-03T05:16:49.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:16:49.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:16:49.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:16:49.856 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:16:49.856 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:16:49.856 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:16:49.856 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:16:49.857 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:16:49.857 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:16:49.857 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:16:49.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:16:49.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:16:49.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:16:49.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:16:49.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:16:49.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:16:49.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:16:49.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:16:49.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:16:49.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:16:49.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:16:49.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:16:49.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-03T05:16:49.858 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:16:49.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=421f5f5d-8bd3-4f31-8c05-74fde83578e1 2024-06-03T05:16:49.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 421f5f5d-8bd3-4f31-8c05-74fde83578e1' 2024-06-03T05:16:49.859 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 421f5f5d-8bd3-4f31-8c05-74fde83578e1 2024-06-03T05:16:49.859 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:16:49.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDBUV1mVi4UNBAAHjVodsF9t8/g6lCqo5n5bg== 2024-06-03T05:16:49.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDBUV1mVi4UNBAAHjVodsF9t8/g6lCqo5n5bg=="}' 2024-06-03T05:16:49.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 421f5f5d-8bd3-4f31-8c05-74fde83578e1 -i td/ceph-helpers/0/new.json 2024-06-03T05:16:50.192 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:16:50.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-03T05:16:50.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDBUV1mVi4UNBAAHjVodsF9t8/g6lCqo5n5bg== --osd-uuid 421f5f5d-8bd3-4f31-8c05-74fde83578e1 2024-06-03T05:16:50.238 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:16:50.234+0000 7f72c46ba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:16:50.252 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:16:50.246+0000 7f72c46ba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:16:50.254 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:16:50.250+0000 7f72c46ba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:16:50.254 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:16:50.250+0000 7f72c46ba7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:16:50.254 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:16:50.250+0000 7f72c46ba7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:16:50.254 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:16:50.250+0000 7f72c46ba7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:16:50.292 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:16:50.286+0000 7f72c46ba7c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-03T05:16:56.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-03T05:16:56.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:16:56.559 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T05:16:56.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T05:16:56.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:16:57.118 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:16:57.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T05:16:57.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:16:57.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:16:57.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:16:57.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:16:57.144 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:16:57.138+0000 7fe77d8db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:16:57.146 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:16:57.142+0000 7fe77d8db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:16:57.148 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:16:57.142+0000 7fe77d8db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:16:57.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T05:16:57.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:16:57.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:16:57.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:16:57.556 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:16:57.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:16:57.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:16:57.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:16:57.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:16:57.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:16:57.702 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:16:57.698+0000 7fe77d8db7c0 -1 Falling back to public interface 2024-06-03T05:16:57.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:16:58.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:16:58.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:16:58.994 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:16:58.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:16:58.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:16:58.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:16:59.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:17:00.392 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:17:00.386+0000 7fe77d8db7c0 -1 osd.0 0 log_to_monitors true 2024-06-03T05:17:00.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:17:00.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:17:00.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:17:00.397 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:17:00.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:17:00.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:17:00.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:17:01.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:17:01.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:17:01.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:17:01.908 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:17:01.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:17:01.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:17:02.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:17:03.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:17:03.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:17:03.350 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:17:03.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:17:03.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:17:03.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:17:03.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:17:04.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:17:04.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:17:04.804 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:17:04.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:17:04.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:17:04.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:17:05.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:17:06.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:17:06.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:17:06.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:17:06.411 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:17:06.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:17:06.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:17:06.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:17:07.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:17:07.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:17:07.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:17:07.817 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:17:07.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:17:07.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:17:08.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:17:09.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:17:09.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:17:09.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:17:09.334 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:17:09.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:17:09.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:17:09.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:17:10.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:17:10.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:17:10.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:17:10.797 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:17:10.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:17:10.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:17:11.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:17:12.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:17:12.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:17:12.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:17:12.292 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:17:12.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:17:12.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:17:12.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:17:13.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:17:13.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:17:13.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:17:13.763 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:17:13.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:17:13.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:17:14.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:17:15.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:17:15.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:17:15.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:17:15.341 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:17:15.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:17:15.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:17:16.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:17:17.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:17:17.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:17:17.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:17:17.155 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:17:17.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:17:17.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:17:17.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:17:18.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:17:18.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:17:18.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:17:18.946 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:17:18.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:17:18.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:17:20.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:17:21.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:17:21.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:17:21.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:17:21.119 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:17:21.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:17:21.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:17:21.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:17:22.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:17:22.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:17:22.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:17:22.897 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:17:22.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:17:22.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:17:23.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:17:24.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:17:24.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:17:24.733 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:17:24.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:17:24.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:17:24.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:17:25.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:17:26.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:17:26.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:17:26.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T05:17:26.190 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-06-03T05:17:26.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:17:26.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:17:26.612 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2296428071,v1:127.0.0.1:6803/2296428071] [v2:127.0.0.1:6804/2296428071,v1:127.0.0.1:6805/2296428071] exists,up 421f5f5d-8bd3-4f31-8c05-74fde83578e1 2024-06-03T05:17:26.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:17:26.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:17:26.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:17:26.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:377: test_kill_daemons: kill_daemons td/ceph-helpers 0 osd 1 2024-06-03T05:17:26.614 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:17:26.614 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:17:26.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:17:26.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:17:26.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:17:27.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 1 2024-06-03T05:17:27.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:381: test_kill_daemons: kill_daemons td/ceph-helpers TERM osd 2024-06-03T05:17:27.625 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:17:27.625 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:17:27.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:17:27.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:17:27.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:17:27.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:17:27.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:385: test_kill_daemons: kill_daemons td/ceph-helpers TERM mgr 2024-06-03T05:17:27.936 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:17:27.936 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:17:27.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:17:27.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:17:27.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:17:28.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:17:28.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:389: test_kill_daemons: kill_daemons td/ceph-helpers TERM 2024-06-03T05:17:28.045 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:17:28.045 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:17:28.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:17:28.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:17:28.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:17:28.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:17:28.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:390: test_kill_daemons: timeout 5 ceph status 2024-06-03T05:17:33.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:391: test_kill_daemons: teardown td/ceph-helpers 2024-06-03T05:17:33.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:17:33.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:17:33.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:17:33.360 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:17:33.360 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:17:33.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:17:33.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:17:33.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:17:33.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:17:33.368 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:17:33.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:17:33.370 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:17:33.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:17:33.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:17:33.371 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:17:33.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:17:33.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:17:33.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:17:33.373 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:17:33.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:17:33.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:17:33.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:17:33.382 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:17:33.382 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:17:33.382 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:17:33.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:17:33.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:17:33.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:17:33.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:17:33.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_objectstore_tool td/ceph-helpers 2024-06-03T05:17:33.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: test_objectstore_tool: local dir=td/ceph-helpers 2024-06-03T05:17:33.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: test_objectstore_tool: setup td/ceph-helpers 2024-06-03T05:17:33.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-03T05:17:33.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-03T05:17:33.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:17:33.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:17:33.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:17:33.385 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:17:33.385 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:17:33.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:17:33.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:17:33.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:17:33.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:17:33.387 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:17:33.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:17:33.388 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:17:33.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:17:33.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:17:33.390 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:17:33.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:17:33.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:17:33.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:17:33.392 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:17:33.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:17:33.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:17:33.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:17:33.395 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:17:33.395 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:17:33.396 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:17:33.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:17:33.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:17:33.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:17:33.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-03T05:17:33.398 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:17:33.398 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:17:33.398 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:17:33.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70897 2024-06-03T05:17:33.401 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:17:33.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:17:33.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:17:33.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-03T05:17:33.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: test_objectstore_tool: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:17:33.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-03T05:17:33.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:17:33.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:17:33.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:17:33.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-03T05:17:33.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:17:33.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:17:33.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:17:33.844 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:17:33.844 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:17:33.844 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:17:33.844 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:17:33.844 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:17:33.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:17:34.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:17:34.164 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:17:34.164 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:17:34.164 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:17:34.164 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:17:34.165 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:17:34.165 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:17:34.165 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:17:34.165 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:17:34.166 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:17:34.166 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:17:34.166 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:17:34.166 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:17:34.166 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:17:34.166 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get fsid 2024-06-03T05:17:34.282 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:17:34.282 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:17:34.282 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:17:34.282 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:17:34.283 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:17:34.284 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:17:34.284 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:17:34.284 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:17:34.284 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:17:34.284 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:17:34.284 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:17:34.284 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:17:34.284 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:17:34.284 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get mon_host 2024-06-03T05:17:34.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1311: test_objectstore_tool: local osd=0 2024-06-03T05:17:34.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: test_objectstore_tool: run_mgr td/ceph-helpers x 2024-06-03T05:17:34.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-03T05:17:34.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T05:17:34.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T05:17:34.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T05:17:34.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-03T05:17:34.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T05:17:35.314 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T05:17:35.315 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:17:35.315 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:17:35.315 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:17:35.315 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:17:35.315 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:17:35.316 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:17:35.316 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:17:35.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:17:35.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: test_objectstore_tool: run_osd td/ceph-helpers 0 2024-06-03T05:17:35.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:17:35.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:17:35.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T05:17:35.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:17:35.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-03T05:17:35.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:17:35.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:17:35.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:17:35.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:17:35.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-03T05:17:35.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-03T05:17:35.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:17:35.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:17:35.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:17:35.355 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:17:35.355 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:17:35.355 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:17:35.355 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:17:35.355 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:17:35.355 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:17:35.355 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:17:35.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:17:35.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:17:35.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:17:35.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:17:35.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:17:35.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:17:35.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:17:35.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:17:35.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:17:35.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:17:35.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:17:35.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:17:35.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-03T05:17:35.356 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:17:35.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f3d1a279-cf51-45a6-8cea-e07bb32f212e 2024-06-03T05:17:35.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f3d1a279-cf51-45a6-8cea-e07bb32f212e' 2024-06-03T05:17:35.357 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 f3d1a279-cf51-45a6-8cea-e07bb32f212e 2024-06-03T05:17:35.357 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:17:35.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDvUV1m8MUuFhAAz9Gd0y+mZuKw1Qj1XE5xUg== 2024-06-03T05:17:35.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDvUV1m8MUuFhAAz9Gd0y+mZuKw1Qj1XE5xUg=="}' 2024-06-03T05:17:35.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f3d1a279-cf51-45a6-8cea-e07bb32f212e -i td/ceph-helpers/0/new.json 2024-06-03T05:17:35.659 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:17:35.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-03T05:17:35.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDvUV1m8MUuFhAAz9Gd0y+mZuKw1Qj1XE5xUg== --osd-uuid f3d1a279-cf51-45a6-8cea-e07bb32f212e 2024-06-03T05:17:35.710 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:17:35.706+0000 7fb0fdeb07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:17:35.728 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:17:35.722+0000 7fb0fdeb07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:17:35.730 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:17:35.726+0000 7fb0fdeb07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:17:35.730 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:17:35.726+0000 7fb0fdeb07c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:17:35.730 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:17:35.726+0000 7fb0fdeb07c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:17:35.730 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:17:35.726+0000 7fb0fdeb07c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:17:35.768 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:17:35.762+0000 7fb0fdeb07c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-03T05:17:42.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-03T05:17:42.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:17:42.119 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T05:17:42.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T05:17:42.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:17:42.668 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:17:42.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T05:17:42.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:17:42.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:17:42.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:17:42.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:17:42.699 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:17:42.693+0000 7f221f7d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:17:42.726 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:17:42.721+0000 7f221f7d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:17:42.727 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:17:42.721+0000 7f221f7d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:17:43.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T05:17:43.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:17:43.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:17:43.089 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:17:43.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:17:43.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:17:43.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:17:43.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:17:43.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:17:43.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:17:43.286 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:17:43.281+0000 7f221f7d67c0 -1 Falling back to public interface 2024-06-03T05:17:43.513 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:17:44.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:17:44.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:17:44.515 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:17:44.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:17:44.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:17:44.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:17:44.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:17:45.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:17:45.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:17:45.931 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:17:45.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:17:45.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:17:45.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:17:45.958 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:17:45.953+0000 7f221f7d67c0 -1 osd.0 0 log_to_monitors true 2024-06-03T05:17:46.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:17:47.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:17:47.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:17:47.397 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:17:47.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:17:47.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:17:47.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:17:47.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:17:48.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:17:48.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:17:48.884 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:17:48.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:17:48.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:17:48.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:17:49.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:17:50.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:17:50.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:17:50.319 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:17:50.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:17:50.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:17:50.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:17:50.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:17:51.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:17:51.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:17:51.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:17:51.876 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:17:51.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:17:51.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:17:52.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:17:53.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:17:53.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:17:53.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:17:53.318 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:17:53.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:17:53.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:17:53.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:17:54.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:17:54.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:17:54.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:17:54.894 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:17:54.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:17:54.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:17:55.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:17:56.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:17:56.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:17:56.338 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:17:56.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:17:56.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:17:56.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:17:56.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:17:57.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:17:57.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:17:57.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:17:57.860 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:17:57.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:17:57.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:17:58.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:17:59.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:17:59.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:17:59.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:17:59.277 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:17:59.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:17:59.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:17:59.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:18:00.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:18:00.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:18:00.746 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:18:00.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:18:00.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:18:00.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:18:01.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:18:02.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:18:02.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:18:02.587 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:18:02.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:18:02.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:18:02.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:18:03.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:18:04.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:18:04.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:18:04.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:18:04.618 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:18:04.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:18:04.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:18:05.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:18:06.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:18:06.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:18:06.037 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:18:06.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:18:06.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:18:06.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:18:06.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:18:07.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:18:07.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:18:07.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:18:07.695 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:18:07.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:18:07.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:18:08.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:18:09.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:18:09.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:18:09.764 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:18:09.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:18:09.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:18:09.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:18:10.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:18:11.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:18:11.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:18:11.455 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-06-03T05:18:11.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T05:18:11.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:18:11.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:18:11.858 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1816821175,v1:127.0.0.1:6803/1816821175] [v2:127.0.0.1:6804/1816821175,v1:127.0.0.1:6805/1816821175] exists,up f3d1a279-cf51-45a6-8cea-e07bb32f212e 2024-06-03T05:18:11.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:18:11.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:18:11.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:18:11.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: test_objectstore_tool: create_rbd_pool 2024-06-03T05:18:11.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-03T05:18:12.305 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' does not exist 2024-06-03T05:18:12.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-03T05:18:12.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-03T05:18:12.903 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' already exists 2024-06-03T05:18:12.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T05:18:13.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-03T05:18:15.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: test_objectstore_tool: wait_for_clean 2024-06-03T05:18:15.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T05:18:15.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T05:18:15.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T05:18:15.263 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T05:18:15.263 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:18:15.264 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:18:15.264 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:18:15.264 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:18:15.264 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:18:15.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T05:18:15.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T05:18:15.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T05:18:15.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T05:18:15.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T05:18:15.384 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T05:18:15.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-06-03T05:18:15.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T05:18:15.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:18:15.806 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T05:18:16.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836489 2024-06-03T05:18:16.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836489 2024-06-03T05:18:16.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489' 2024-06-03T05:18:16.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:18:16.003 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836489 2024-06-03T05:18:16.003 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:18:16.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T05:18:16.004 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836489 2024-06-03T05:18:16.004 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:18:16.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836489 2024-06-03T05:18:16.005 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836489 2024-06-03T05:18:16.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2024-06-03T05:18:16.005 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:18:16.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836487 -lt 21474836489 2024-06-03T05:18:16.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T05:18:17.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T05:18:17.426 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:18:17.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836487 -lt 21474836489 2024-06-03T05:18:17.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T05:18:18.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-03T05:18:18.906 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:18:19.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836490 -lt 21474836489 2024-06-03T05:18:19.325 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T05:18:19.325 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:18:19.326 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:18:19.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-03T05:18:19.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T05:18:19.882 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T05:18:19.882 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T05:18:19.882 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T05:18:19.882 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T05:18:19.882 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T05:18:19.882 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T05:18:20.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T05:18:20.519 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T05:18:20.519 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:18:20.519 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:18:21.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-03T05:18:21.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T05:18:21.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T05:18:21.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1316: test_objectstore_tool: rados --pool rbd put GROUP /etc/group 2024-06-03T05:18:21.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: test_objectstore_tool: objectstore_tool td/ceph-helpers 0 GROUP get-bytes 2024-06-03T05:18:21.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: test_objectstore_tool: diff - /etc/group 2024-06-03T05:18:21.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/ceph-helpers 2024-06-03T05:18:21.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T05:18:21.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-03T05:18:21.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T05:18:21.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/ceph-helpers 0 GROUP get-bytes 2024-06-03T05:18:21.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/ceph-helpers 2024-06-03T05:18:21.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T05:18:21.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-03T05:18:21.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T05:18:21.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/ceph-helpers TERM osd.0 2024-06-03T05:18:21.174 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:18:21.174 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:18:21.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:18:21.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:18:21.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:18:21.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:18:21.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/ceph-helpers 0 GROUP get-bytes 2024-06-03T05:18:21.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/ceph-helpers 2024-06-03T05:18:21.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T05:18:21.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T05:18:21.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T05:18:21.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/ceph-helpers/0 2024-06-03T05:18:21.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/ceph-helpers/0 GROUP get-bytes 2024-06-03T05:18:23.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/ceph-helpers 0 2024-06-03T05:18:23.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-06-03T05:18:23.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T05:18:23.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T05:18:23.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T05:18:23.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-06-03T05:18:23.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:18:23.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:18:23.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:18:23.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:18:23.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-03T05:18:23.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-03T05:18:23.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T05:18:23.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T05:18:23.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:18:23.442 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T05:18:23.442 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:18:23.442 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:18:23.442 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:18:23.442 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:18:23.442 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:18:23.443 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:18:23.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:18:23.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:18:23.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:18:23.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:18:23.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:18:23.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:18:23.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:18:23.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:18:23.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T05:18:23.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T05:18:23.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-06-03T05:18:23.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T05:18:23.445 INFO:tasks.workunit.client.0.smithi088.stderr:start osd.0 2024-06-03T05:18:23.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:18:23.445 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-06-03T05:18:23.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T05:18:23.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T05:18:23.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T05:18:23.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T05:18:23.482 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:18:23.477+0000 7f4c1109c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:18:23.485 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:18:23.481+0000 7f4c1109c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:18:23.486 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:18:23.481+0000 7f4c1109c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:18:23.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T05:18:23.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:18:23.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:18:23.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:18:23.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:18:23.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:18:23.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:18:23.845 INFO:tasks.workunit.client.0.smithi088.stderr:0 2024-06-03T05:18:23.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:18:23.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:18:24.030 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:18:24.025+0000 7f4c1109c7c0 -1 Falling back to public interface 2024-06-03T05:18:24.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:18:25.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:18:25.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:18:25.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:18:25.285 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-06-03T05:18:25.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:18:25.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:18:25.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:18:26.652 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:18:26.645+0000 7f4c1109c7c0 -1 osd.0 14 log_to_monitors true 2024-06-03T05:18:26.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:18:26.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:18:26.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:18:26.707 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-06-03T05:18:26.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:18:26.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:18:27.156 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 up in weight 1 up_from 17 up_thru 7 down_at 15 last_clean_interval [5,14) [v2:127.0.0.1:6802/1408693477,v1:127.0.0.1:6803/1408693477] [v2:127.0.0.1:6804/1408693477,v1:127.0.0.1:6805/1408693477] exists,up f3d1a279-cf51-45a6-8cea-e07bb32f212e 2024-06-03T05:18:27.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:18:27.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:18:27.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:18:27.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-03T05:18:27.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T05:18:27.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T05:18:27.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T05:18:27.158 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T05:18:27.158 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:18:27.158 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:18:27.158 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:18:27.158 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:18:27.158 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:18:27.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T05:18:27.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T05:18:27.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T05:18:27.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T05:18:27.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T05:18:27.248 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T05:18:27.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-06-03T05:18:27.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T05:18:27.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:18:27.695 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T05:18:27.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=73014444035 2024-06-03T05:18:27.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 73014444035 2024-06-03T05:18:27.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-73014444035' 2024-06-03T05:18:27.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:18:27.881 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-73014444035 2024-06-03T05:18:27.881 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:18:27.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T05:18:27.882 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-73014444035 2024-06-03T05:18:27.882 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:18:27.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=73014444035 2024-06-03T05:18:27.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 73014444035' 2024-06-03T05:18:27.884 INFO:tasks.workunit.client.0.smithi088.stderr:waiting osd.0 seq 73014444035 2024-06-03T05:18:27.884 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:18:28.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 73014444035 -lt 73014444035 2024-06-03T05:18:28.296 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T05:18:28.296 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:18:28.297 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:18:28.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-03T05:18:28.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T05:18:28.860 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T05:18:28.860 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T05:18:28.860 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T05:18:28.860 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T05:18:28.860 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T05:18:28.861 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T05:18:29.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T05:18:29.306 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T05:18:29.307 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:18:29.307 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:18:29.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-03T05:18:29.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T05:18:29.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T05:18:29.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1319: test_objectstore_tool: objectstore_tool td/ceph-helpers 0 NOTEXISTS get-bytes 2024-06-03T05:18:29.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/ceph-helpers 2024-06-03T05:18:29.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-03T05:18:29.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-03T05:18:29.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-03T05:18:29.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/ceph-helpers 0 NOTEXISTS get-bytes 2024-06-03T05:18:29.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/ceph-helpers 2024-06-03T05:18:29.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-03T05:18:29.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-03T05:18:29.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-03T05:18:29.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/ceph-helpers TERM osd.0 2024-06-03T05:18:29.862 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:18:29.862 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:18:29.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:18:29.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:18:29.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:18:30.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:18:30.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/ceph-helpers 0 NOTEXISTS get-bytes 2024-06-03T05:18:30.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/ceph-helpers 2024-06-03T05:18:30.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-03T05:18:30.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-03T05:18:30.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-03T05:18:30.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/ceph-helpers/0 2024-06-03T05:18:30.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/ceph-helpers/0 NOTEXISTS get-bytes 2024-06-03T05:18:31.207 INFO:tasks.workunit.client.0.smithi088.stderr:No object id 'NOTEXISTS' found or invalid JSON specified 2024-06-03T05:18:31.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2024-06-03T05:18:31.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: return 1 2024-06-03T05:18:31.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: return 1 2024-06-03T05:18:31.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1320: test_objectstore_tool: teardown td/ceph-helpers 2024-06-03T05:18:31.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:18:31.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:18:31.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:18:31.745 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:18:31.745 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:18:31.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:18:31.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:18:31.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:18:31.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:18:31.855 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:18:31.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:18:31.856 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:18:31.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:18:31.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:18:31.857 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:18:31.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:18:31.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:18:31.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:18:31.859 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:18:31.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:18:31.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:18:31.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:18:31.869 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:18:31.870 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:18:31.870 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:18:31.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:18:31.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:18:31.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:18:31.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:18:31.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_pg_schedule_scrub td/ceph-helpers 2024-06-03T05:18:31.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1931: test_pg_schedule_scrub: local dir=td/ceph-helpers 2024-06-03T05:18:31.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1933: test_pg_schedule_scrub: setup td/ceph-helpers 2024-06-03T05:18:31.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-03T05:18:31.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-03T05:18:31.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:18:31.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:18:31.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:18:31.872 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:18:31.873 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:18:31.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:18:31.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:18:31.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:18:31.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:18:31.874 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:18:31.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:18:31.875 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:18:31.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:18:31.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:18:31.876 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:18:31.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:18:31.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:18:31.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:18:31.878 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:18:31.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:18:31.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:18:31.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:18:31.881 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:18:31.881 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:18:31.881 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:18:31.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:18:31.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:18:31.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:18:31.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-03T05:18:31.884 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:18:31.884 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:18:31.884 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:18:31.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70897 2024-06-03T05:18:31.886 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:18:31.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:18:31.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:18:31.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-03T05:18:31.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: test_pg_schedule_scrub: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:18:31.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-03T05:18:31.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:18:31.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:18:31.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:18:31.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-03T05:18:31.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:18:32.274 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:18:32.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:18:32.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:18:32.275 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:18:32.275 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:18:32.275 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:18:32.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:18:32.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:18:32.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:18:32.519 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:18:32.519 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:18:32.519 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:18:32.519 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:18:32.520 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:18:32.520 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:18:32.520 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:18:32.520 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:18:32.521 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:18:32.521 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:18:32.521 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:18:32.521 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:18:32.521 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:18:32.521 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get fsid 2024-06-03T05:18:32.632 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:18:32.632 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:18:32.632 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:18:32.632 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:18:32.633 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:18:32.633 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:18:32.634 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:18:32.634 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:18:32.634 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:18:32.634 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:18:32.634 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:18:32.634 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:18:32.634 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:18:32.634 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get mon_host 2024-06-03T05:18:32.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: test_pg_schedule_scrub: run_mgr td/ceph-helpers x 2024-06-03T05:18:32.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-03T05:18:32.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T05:18:32.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T05:18:32.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T05:18:32.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-03T05:18:32.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T05:18:33.634 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T05:18:33.634 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:18:33.634 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:18:33.634 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:18:33.634 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:18:33.634 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:18:33.635 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:18:33.635 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:18:33.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:18:33.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: test_pg_schedule_scrub: run_osd td/ceph-helpers 0 2024-06-03T05:18:33.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:18:33.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:18:33.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T05:18:33.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:18:33.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-03T05:18:33.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:18:33.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:18:33.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:18:33.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:18:33.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-03T05:18:33.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-03T05:18:33.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:18:33.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:18:33.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:18:33.669 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:18:33.669 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:18:33.669 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:18:33.670 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:18:33.670 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:18:33.670 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:18:33.670 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:18:33.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:18:33.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:18:33.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:18:33.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:18:33.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:18:33.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:18:33.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:18:33.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:18:33.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:18:33.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:18:33.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:18:33.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:18:33.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-03T05:18:33.671 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:18:33.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9abc5fbc-6aaf-47c3-b77a-64ce0870d012 2024-06-03T05:18:33.671 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 9abc5fbc-6aaf-47c3-b77a-64ce0870d012 2024-06-03T05:18:33.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9abc5fbc-6aaf-47c3-b77a-64ce0870d012' 2024-06-03T05:18:33.672 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:18:33.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQApUl1miP/tKBAA92jFVEu80h+AUT4taeLEug== 2024-06-03T05:18:33.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQApUl1miP/tKBAA92jFVEu80h+AUT4taeLEug=="}' 2024-06-03T05:18:33.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9abc5fbc-6aaf-47c3-b77a-64ce0870d012 -i td/ceph-helpers/0/new.json 2024-06-03T05:18:33.998 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:18:34.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-03T05:18:34.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQApUl1miP/tKBAA92jFVEu80h+AUT4taeLEug== --osd-uuid 9abc5fbc-6aaf-47c3-b77a-64ce0870d012 2024-06-03T05:18:34.044 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:18:34.037+0000 7f9b2d7d47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:18:34.084 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:18:34.077+0000 7f9b2d7d47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:18:34.085 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:18:34.081+0000 7f9b2d7d47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:18:34.086 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:18:34.081+0000 7f9b2d7d47c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:18:34.086 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:18:34.081+0000 7f9b2d7d47c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:18:34.086 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:18:34.081+0000 7f9b2d7d47c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:18:34.132 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:18:34.125+0000 7f9b2d7d47c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-03T05:18:40.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-03T05:18:40.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:18:40.657 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T05:18:40.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T05:18:40.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:18:41.213 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:18:41.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T05:18:41.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:18:41.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:18:41.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:18:41.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:18:41.246 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:18:41.241+0000 7fde4c2357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:18:41.248 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:18:41.241+0000 7fde4c2357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:18:41.249 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:18:41.245+0000 7fde4c2357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:18:41.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T05:18:41.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:18:41.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:18:41.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:18:41.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:18:41.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:18:41.653 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:18:41.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:18:41.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:18:41.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:18:41.831 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:18:41.825+0000 7fde4c2357c0 -1 Falling back to public interface 2024-06-03T05:18:42.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:18:43.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:18:43.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:18:43.064 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:18:43.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:18:43.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:18:43.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:18:43.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:18:44.391 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:18:44.385+0000 7fde4c2357c0 -1 osd.0 0 log_to_monitors true 2024-06-03T05:18:44.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:18:44.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:18:44.493 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:18:44.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:18:44.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:18:44.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:18:45.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:18:46.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:18:46.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:18:46.102 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:18:46.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:18:46.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:18:46.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:18:46.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:18:47.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:18:47.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:18:47.566 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:18:47.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:18:47.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:18:47.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:18:47.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:18:48.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:18:48.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:18:48.989 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:18:48.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:18:48.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:18:48.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:18:49.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:18:50.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:18:50.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:18:50.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:18:50.562 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:18:50.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:18:50.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:18:51.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:18:52.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:18:52.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:18:52.048 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:18:52.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:18:52.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:18:52.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:18:52.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:18:53.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:18:53.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:18:53.594 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:18:53.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:18:53.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:18:53.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:18:54.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:18:55.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:18:55.057 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:18:55.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:18:55.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:18:55.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:18:55.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:18:55.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:18:56.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:18:56.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:18:56.537 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:18:56.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:18:56.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:18:56.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:18:57.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:18:58.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:18:58.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:18:58.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:18:58.007 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:18:58.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:18:58.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:18:59.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:19:00.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:19:00.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:19:00.217 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:19:00.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:19:00.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:19:00.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:19:01.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:19:02.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:19:02.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:19:02.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:19:02.469 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:19:02.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:19:02.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:19:03.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:19:04.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:19:04.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:19:04.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:19:04.483 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:19:04.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:19:04.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:19:05.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:19:06.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:19:06.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:19:06.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:19:06.380 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:19:06.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:19:06.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:19:07.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:19:08.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:19:08.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:19:08.657 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:19:08.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:19:08.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:19:08.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:19:09.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:19:10.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:19:10.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:19:10.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:19:10.229 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:19:10.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:19:10.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:19:10.704 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/674980614,v1:127.0.0.1:6803/674980614] [v2:127.0.0.1:6804/674980614,v1:127.0.0.1:6805/674980614] exists,up 9abc5fbc-6aaf-47c3-b77a-64ce0870d012 2024-06-03T05:19:10.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:19:10.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:19:10.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:19:10.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: test_pg_schedule_scrub: create_rbd_pool 2024-06-03T05:19:10.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-03T05:19:11.126 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' does not exist 2024-06-03T05:19:11.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-03T05:19:11.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-03T05:19:11.711 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' already exists 2024-06-03T05:19:11.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T05:19:12.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-03T05:19:13.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1938: test_pg_schedule_scrub: wait_for_clean 2024-06-03T05:19:13.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T05:19:13.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T05:19:13.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T05:19:13.862 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T05:19:13.862 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:19:13.863 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:19:13.863 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:19:13.863 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:19:13.863 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:19:13.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T05:19:13.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T05:19:13.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T05:19:13.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T05:19:13.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T05:19:13.979 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T05:19:14.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-06-03T05:19:14.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T05:19:14.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:19:14.413 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T05:19:14.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836488 2024-06-03T05:19:14.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836488 2024-06-03T05:19:14.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836488' 2024-06-03T05:19:14.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:19:14.584 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836488 2024-06-03T05:19:14.584 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:19:14.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T05:19:14.585 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836488 2024-06-03T05:19:14.585 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:19:14.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836488 2024-06-03T05:19:14.586 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836488 2024-06-03T05:19:14.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2024-06-03T05:19:14.587 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:19:14.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836486 -lt 21474836488 2024-06-03T05:19:14.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T05:19:15.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T05:19:15.994 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:19:16.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836489 -lt 21474836488 2024-06-03T05:19:16.398 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T05:19:16.399 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:19:16.399 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:19:16.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-03T05:19:16.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T05:19:16.989 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T05:19:16.989 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T05:19:16.989 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T05:19:16.989 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T05:19:16.990 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T05:19:16.990 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T05:19:17.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T05:19:17.445 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T05:19:17.445 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:19:17.445 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:19:18.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-03T05:19:18.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T05:19:18.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T05:19:18.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1939: test_pg_schedule_scrub: pg_schedule_scrub 1.0 2024-06-03T05:19:18.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1917: pg_schedule_scrub: local pgid=1.0 2024-06-03T05:19:18.024 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: pg_schedule_scrub: get_last_scrub_stamp 1.0 2024-06-03T05:19:18.024 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T05:19:18.024 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T05:19:18.024 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T05:19:18.024 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T05:19:18.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: pg_schedule_scrub: local last_scrub=2024-06-03T05:19:11.536431+0000 2024-06-03T05:19:18.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1919: pg_schedule_scrub: ceph pg scrub 1.0 2024-06-03T05:19:18.845 INFO:tasks.workunit.client.0.smithi088.stderr:instructing pg 1.0 on osd.0 to scrub 2024-06-03T05:19:18.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1920: pg_schedule_scrub: wait_for_scrub 1.0 2024-06-03T05:19:11.536431+0000 2024-06-03T05:19:18.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-06-03T05:19:18.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T05:19:11.536431+0000 2024-06-03T05:19:18.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T05:19:18.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T05:19:18.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T05:19:18.862 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T05:19:18.863 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T05:19:18.863 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T05:19:18.863 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T05:19:18.863 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T05:19:19.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T05:19:11.536431+0000 '>' 2024-06-03T05:19:11.536431+0000 2024-06-03T05:19:19.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T05:19:20.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T05:19:20.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T05:19:20.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T05:19:20.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T05:19:20.247 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T05:19:20.247 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T05:19:20.247 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T05:19:20.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T05:19:19.602395+0000 '>' 2024-06-03T05:19:11.536431+0000 2024-06-03T05:19:20.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T05:19:20.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1940: test_pg_schedule_scrub: kill_daemons td/ceph-helpers KILL osd 2024-06-03T05:19:20.645 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:19:20.646 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:19:20.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:19:20.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:19:20.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:19:20.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:19:20.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1941: test_pg_schedule_scrub: TIMEOUT=1 2024-06-03T05:19:20.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1941: test_pg_schedule_scrub: pg_scrub 1.0 2024-06-03T05:19:20.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=1.0 2024-06-03T05:19:20.752 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 1.0 2024-06-03T05:19:20.752 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T05:19:20.752 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T05:19:20.753 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T05:19:20.753 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T05:19:21.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-06-03T05:19:19.602395+0000 2024-06-03T05:19:21.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 1.0 2024-06-03T05:19:21.540 INFO:tasks.workunit.client.0.smithi088.stderr:instructing pg 1.0 on osd.0 to scrub 2024-06-03T05:19:21.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 1.0 2024-06-03T05:19:19.602395+0000 2024-06-03T05:19:21.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-06-03T05:19:21.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T05:19:19.602395+0000 2024-06-03T05:19:21.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T05:19:21.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T05:19:21.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 1 )) 2024-06-03T05:19:21.557 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T05:19:21.557 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T05:19:21.557 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T05:19:21.557 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T05:19:21.557 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T05:19:21.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T05:19:19.602395+0000 '>' 2024-06-03T05:19:19.602395+0000 2024-06-03T05:19:21.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T05:19:22.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T05:19:22.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 1 )) 2024-06-03T05:19:22.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: return 1 2024-06-03T05:19:22.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1942: test_pg_schedule_scrub: teardown td/ceph-helpers 2024-06-03T05:19:22.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:19:22.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:19:22.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:19:22.960 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:19:22.960 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:19:22.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:19:22.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:19:22.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:19:23.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:19:23.070 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:19:23.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:19:23.071 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:19:23.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:19:23.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:19:23.073 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:19:23.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:19:23.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:19:23.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:19:23.074 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:19:23.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:19:23.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:19:23.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:19:23.083 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:19:23.083 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:19:23.083 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:19:23.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:19:23.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:19:23.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:19:23.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:19:23.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_pg_scrub td/ceph-helpers 2024-06-03T05:19:23.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: test_pg_scrub: local dir=td/ceph-helpers 2024-06-03T05:19:23.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: test_pg_scrub: setup td/ceph-helpers 2024-06-03T05:19:23.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-03T05:19:23.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-03T05:19:23.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:19:23.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:19:23.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:19:23.086 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:19:23.086 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:19:23.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:19:23.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:19:23.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:19:23.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:19:23.089 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:19:23.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:19:23.094 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:19:23.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:19:23.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:19:23.095 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:19:23.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:19:23.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:19:23.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:19:23.095 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:19:23.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:19:23.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:19:23.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:19:23.096 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:19:23.096 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:19:23.096 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:19:23.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:19:23.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:19:23.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:19:23.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-03T05:19:23.099 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:19:23.099 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:19:23.099 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:19:23.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70897 2024-06-03T05:19:23.101 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:19:23.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:19:23.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:19:23.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-03T05:19:23.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: test_pg_scrub: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:19:23.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-03T05:19:23.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:19:23.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:19:23.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:19:23.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-03T05:19:23.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:19:23.527 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:19:23.527 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:19:23.527 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:19:23.528 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:19:23.528 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:19:23.528 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:19:23.528 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:19:23.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:19:23.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:19:23.782 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:19:23.782 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:19:23.782 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:19:23.782 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:19:23.783 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:19:23.783 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:19:23.783 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:19:23.783 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:19:23.784 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:19:23.784 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:19:23.784 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:19:23.784 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:19:23.784 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:19:23.784 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get fsid 2024-06-03T05:19:23.896 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:19:23.896 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:19:23.896 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:19:23.896 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:19:23.897 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:19:23.897 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:19:23.898 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:19:23.898 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:19:23.898 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:19:23.898 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:19:23.898 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:19:23.898 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:19:23.898 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:19:23.899 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get mon_host 2024-06-03T05:19:23.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1893: test_pg_scrub: run_mgr td/ceph-helpers x 2024-06-03T05:19:23.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-03T05:19:23.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T05:19:23.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T05:19:23.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T05:19:23.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-03T05:19:23.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T05:19:24.894 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T05:19:24.895 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:19:24.895 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:19:24.895 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:19:24.895 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:19:24.896 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:19:24.896 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:19:24.896 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:19:24.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:19:24.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1894: test_pg_scrub: run_osd td/ceph-helpers 0 2024-06-03T05:19:24.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:19:24.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:19:24.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T05:19:24.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:19:24.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-03T05:19:24.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:19:24.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:19:24.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:19:24.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:19:24.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-03T05:19:24.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-03T05:19:24.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:19:24.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:19:24.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:19:24.935 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:19:24.935 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:19:24.935 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:19:24.935 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:19:24.935 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:19:24.935 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:19:24.935 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:19:24.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:19:24.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:19:24.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:19:24.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:19:24.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:19:24.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:19:24.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:19:24.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:19:24.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:19:24.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:19:24.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:19:24.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:19:24.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-03T05:19:24.937 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:19:24.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f44b4335-ae89-4afe-a1dd-5d09966afcd2 2024-06-03T05:19:24.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f44b4335-ae89-4afe-a1dd-5d09966afcd2' 2024-06-03T05:19:24.937 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 f44b4335-ae89-4afe-a1dd-5d09966afcd2 2024-06-03T05:19:24.938 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:19:24.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBcUl1mGeSxOBAAPUDKNxneAAyBp4Q2NPJ5tg== 2024-06-03T05:19:24.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBcUl1mGeSxOBAAPUDKNxneAAyBp4Q2NPJ5tg=="}' 2024-06-03T05:19:24.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f44b4335-ae89-4afe-a1dd-5d09966afcd2 -i td/ceph-helpers/0/new.json 2024-06-03T05:19:25.243 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:19:25.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-03T05:19:25.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBcUl1mGeSxOBAAPUDKNxneAAyBp4Q2NPJ5tg== --osd-uuid f44b4335-ae89-4afe-a1dd-5d09966afcd2 2024-06-03T05:19:25.294 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:19:25.289+0000 7f95b5d017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:19:25.312 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:19:25.305+0000 7f95b5d017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:19:25.313 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:19:25.309+0000 7f95b5d017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:19:25.313 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:19:25.309+0000 7f95b5d017c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:19:25.313 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:19:25.309+0000 7f95b5d017c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:19:25.314 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:19:25.309+0000 7f95b5d017c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:19:25.352 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:19:25.345+0000 7f95b5d017c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-03T05:19:31.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-03T05:19:31.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:19:31.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T05:19:31.840 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T05:19:31.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:19:32.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T05:19:32.370 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:19:32.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:19:32.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:19:32.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:19:32.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:19:32.396 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:19:32.389+0000 7f6ee5b807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:19:32.398 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:19:32.393+0000 7f6ee5b807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:19:32.399 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:19:32.393+0000 7f6ee5b807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:19:32.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T05:19:32.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:19:32.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:19:32.804 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:19:32.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:19:32.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:19:32.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:19:32.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:19:32.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:19:32.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:19:32.954 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:19:32.949+0000 7f6ee5b807c0 -1 Falling back to public interface 2024-06-03T05:19:33.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:19:34.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:19:34.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:19:34.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:19:34.203 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:19:34.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:19:34.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:19:34.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:19:35.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:19:35.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:19:35.607 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:19:35.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:19:35.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:19:35.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:19:35.639 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:19:35.633+0000 7f6ee5b807c0 -1 osd.0 0 log_to_monitors true 2024-06-03T05:19:36.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:19:37.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:19:37.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:19:37.124 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:19:37.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:19:37.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:19:37.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:19:37.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:19:38.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:19:38.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:19:38.565 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:19:38.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:19:38.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:19:38.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:19:39.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:19:40.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:19:40.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:19:40.018 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:19:40.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:19:40.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:19:40.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:19:40.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:19:41.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:19:41.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:19:41.590 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:19:41.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:19:41.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:19:41.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:19:42.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:19:43.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:19:43.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:19:43.190 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:19:43.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:19:43.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:19:43.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:19:43.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:19:44.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:19:44.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:19:44.622 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:19:44.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:19:44.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:19:44.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:19:45.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:19:46.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:19:46.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:19:46.086 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:19:46.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:19:46.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:19:46.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:19:46.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:19:47.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:19:47.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:19:47.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:19:47.563 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:19:47.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:19:47.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:19:47.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:19:48.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:19:48.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:19:48.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:19:48.989 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:19:48.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:19:48.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:19:49.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:19:50.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:19:50.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:19:50.533 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:19:50.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:19:50.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:19:50.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:19:51.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:19:52.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:19:52.619 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:19:52.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:19:52.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:19:52.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:19:52.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:19:53.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:19:54.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:19:54.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:19:54.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:19:54.640 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:19:54.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:19:54.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:19:55.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:19:56.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:19:56.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:19:56.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:19:56.458 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:19:56.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:19:56.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:19:57.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:19:58.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:19:58.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:19:58.205 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:19:58.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:19:58.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:19:58.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:19:59.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:20:00.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:20:00.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:20:00.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:20:00.313 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:20:00.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:20:00.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:20:00.820 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3896633232,v1:127.0.0.1:6803/3896633232] [v2:127.0.0.1:6804/3896633232,v1:127.0.0.1:6805/3896633232] exists,up f44b4335-ae89-4afe-a1dd-5d09966afcd2 2024-06-03T05:20:00.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:20:00.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:20:00.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:20:00.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1895: test_pg_scrub: create_rbd_pool 2024-06-03T05:20:00.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-03T05:20:01.267 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' does not exist 2024-06-03T05:20:01.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-03T05:20:01.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-03T05:20:01.857 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' already exists 2024-06-03T05:20:01.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T05:20:02.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-03T05:20:04.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1896: test_pg_scrub: wait_for_clean 2024-06-03T05:20:04.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T05:20:04.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T05:20:04.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T05:20:04.116 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T05:20:04.117 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:20:04.117 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:20:04.117 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:20:04.117 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:20:04.117 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:20:04.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T05:20:04.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T05:20:04.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T05:20:04.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T05:20:04.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T05:20:04.240 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T05:20:04.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-06-03T05:20:04.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T05:20:04.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:20:04.657 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T05:20:04.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836489 2024-06-03T05:20:04.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836489 2024-06-03T05:20:04.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489' 2024-06-03T05:20:04.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:20:04.819 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836489 2024-06-03T05:20:04.819 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:20:04.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T05:20:04.820 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836489 2024-06-03T05:20:04.821 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:20:04.821 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836489 2024-06-03T05:20:04.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836489 2024-06-03T05:20:04.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2024-06-03T05:20:04.822 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:20:05.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836487 -lt 21474836489 2024-06-03T05:20:05.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T05:20:06.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T05:20:06.284 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:20:06.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836489 -lt 21474836489 2024-06-03T05:20:06.682 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T05:20:06.682 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:20:06.683 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:20:07.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-03T05:20:07.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T05:20:07.258 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T05:20:07.258 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T05:20:07.258 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T05:20:07.258 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T05:20:07.258 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T05:20:07.258 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T05:20:07.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T05:20:07.701 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T05:20:07.702 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:20:07.702 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:20:08.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-03T05:20:08.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T05:20:08.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T05:20:08.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1897: test_pg_scrub: pg_scrub 1.0 2024-06-03T05:20:08.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=1.0 2024-06-03T05:20:08.231 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 1.0 2024-06-03T05:20:08.231 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T05:20:08.231 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T05:20:08.231 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T05:20:08.231 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T05:20:08.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-06-03T05:20:01.691518+0000 2024-06-03T05:20:08.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 1.0 2024-06-03T05:20:09.030 INFO:tasks.workunit.client.0.smithi088.stderr:instructing pg 1.0 on osd.0 to scrub 2024-06-03T05:20:09.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 1.0 2024-06-03T05:20:01.691518+0000 2024-06-03T05:20:09.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-06-03T05:20:09.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T05:20:01.691518+0000 2024-06-03T05:20:09.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T05:20:09.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T05:20:09.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T05:20:09.047 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T05:20:09.047 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T05:20:09.047 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T05:20:09.047 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T05:20:09.047 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T05:20:09.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T05:20:01.691518+0000 '>' 2024-06-03T05:20:01.691518+0000 2024-06-03T05:20:09.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T05:20:10.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T05:20:10.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T05:20:10.434 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T05:20:10.434 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T05:20:10.435 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T05:20:10.435 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T05:20:10.435 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T05:20:10.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T05:20:01.691518+0000 '>' 2024-06-03T05:20:01.691518+0000 2024-06-03T05:20:10.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T05:20:11.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T05:20:11.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T05:20:11.835 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T05:20:11.835 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T05:20:11.835 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T05:20:11.835 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T05:20:11.835 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T05:20:12.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T05:20:09.658216+0000 '>' 2024-06-03T05:20:01.691518+0000 2024-06-03T05:20:12.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T05:20:12.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1898: test_pg_scrub: kill_daemons td/ceph-helpers KILL osd 2024-06-03T05:20:12.254 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:20:12.255 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:20:12.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:20:12.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:20:12.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:20:12.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:20:12.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1899: test_pg_scrub: TIMEOUT=1 2024-06-03T05:20:12.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1899: test_pg_scrub: pg_scrub 1.0 2024-06-03T05:20:12.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=1.0 2024-06-03T05:20:12.361 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 1.0 2024-06-03T05:20:12.361 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T05:20:12.361 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T05:20:12.362 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T05:20:12.362 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T05:20:12.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-06-03T05:20:09.658216+0000 2024-06-03T05:20:12.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 1.0 2024-06-03T05:20:13.120 INFO:tasks.workunit.client.0.smithi088.stderr:instructing pg 1.0 on osd.0 to scrub 2024-06-03T05:20:13.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 1.0 2024-06-03T05:20:09.658216+0000 2024-06-03T05:20:13.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-06-03T05:20:13.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T05:20:09.658216+0000 2024-06-03T05:20:13.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T05:20:13.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T05:20:13.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 1 )) 2024-06-03T05:20:13.137 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T05:20:13.137 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T05:20:13.137 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T05:20:13.137 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T05:20:13.137 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T05:20:13.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T05:20:09.658216+0000 '>' 2024-06-03T05:20:09.658216+0000 2024-06-03T05:20:13.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T05:20:14.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T05:20:14.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 1 )) 2024-06-03T05:20:14.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: return 1 2024-06-03T05:20:14.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1900: test_pg_scrub: teardown td/ceph-helpers 2024-06-03T05:20:14.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:20:14.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:20:14.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:20:14.534 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:20:14.534 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:20:14.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:20:14.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:20:14.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:20:14.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:20:14.643 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:20:14.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:20:14.644 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:20:14.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:20:14.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:20:14.645 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:20:14.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:20:14.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:20:14.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:20:14.647 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:20:14.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:20:14.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:20:14.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:20:14.655 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:20:14.655 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:20:14.655 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:20:14.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:20:14.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:20:14.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:20:14.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:20:14.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_repair td/ceph-helpers 2024-06-03T05:20:14.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: test_repair: local dir=td/ceph-helpers 2024-06-03T05:20:14.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: test_repair: setup td/ceph-helpers 2024-06-03T05:20:14.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-03T05:20:14.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-03T05:20:14.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:20:14.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:20:14.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:20:14.659 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:20:14.659 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:20:14.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:20:14.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:20:14.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:20:14.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:20:14.660 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:20:14.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:20:14.661 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:20:14.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:20:14.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:20:14.662 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:20:14.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:20:14.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:20:14.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:20:14.664 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:20:14.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:20:14.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:20:14.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:20:14.667 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:20:14.667 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:20:14.667 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:20:14.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:20:14.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:20:14.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:20:14.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-03T05:20:14.670 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:20:14.670 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:20:14.670 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:20:14.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70897 2024-06-03T05:20:14.671 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:20:14.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:20:14.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:20:14.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-03T05:20:14.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: test_repair: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:20:14.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-03T05:20:14.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:20:14.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:20:14.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:20:14.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-03T05:20:14.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:20:15.076 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:20:15.076 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:20:15.076 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:20:15.077 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:20:15.077 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:20:15.077 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:20:15.077 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:20:15.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:20:15.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:20:15.297 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:20:15.297 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:20:15.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:20:15.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:20:15.299 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:20:15.299 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:20:15.299 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:20:15.299 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:20:15.299 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:20:15.299 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:20:15.299 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:20:15.299 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:20:15.300 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:20:15.300 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get fsid 2024-06-03T05:20:15.416 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:20:15.416 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:20:15.416 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:20:15.416 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:20:15.418 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:20:15.418 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:20:15.418 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:20:15.418 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:20:15.418 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:20:15.418 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:20:15.418 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:20:15.418 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:20:15.418 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:20:15.419 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get mon_host 2024-06-03T05:20:15.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: test_repair: run_mgr td/ceph-helpers x 2024-06-03T05:20:15.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-03T05:20:15.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T05:20:15.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T05:20:15.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T05:20:15.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-03T05:20:15.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T05:20:16.435 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T05:20:16.435 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:20:16.436 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:20:16.436 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:20:16.436 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:20:16.436 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:20:16.436 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:20:16.437 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:20:16.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:20:16.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1851: test_repair: run_osd td/ceph-helpers 0 2024-06-03T05:20:16.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:20:16.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:20:16.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T05:20:16.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:20:16.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-03T05:20:16.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:20:16.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:20:16.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:20:16.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:20:16.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-03T05:20:16.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-03T05:20:16.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:20:16.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:20:16.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:20:16.477 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:20:16.477 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:20:16.477 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:20:16.477 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:20:16.477 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:20:16.477 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:20:16.477 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:20:16.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:20:16.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:20:16.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:20:16.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:20:16.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:20:16.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:20:16.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:20:16.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:20:16.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:20:16.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:20:16.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:20:16.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:20:16.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-03T05:20:16.479 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:20:16.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5acf6b98-9c70-4fb2-a537-4311095ba06f 2024-06-03T05:20:16.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 5acf6b98-9c70-4fb2-a537-4311095ba06f' 2024-06-03T05:20:16.479 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 5acf6b98-9c70-4fb2-a537-4311095ba06f 2024-06-03T05:20:16.480 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:20:16.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCQUl1mym50HRAAj0zNEfoRsKq4KJU4JdI5cw== 2024-06-03T05:20:16.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCQUl1mym50HRAAj0zNEfoRsKq4KJU4JdI5cw=="}' 2024-06-03T05:20:16.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5acf6b98-9c70-4fb2-a537-4311095ba06f -i td/ceph-helpers/0/new.json 2024-06-03T05:20:16.809 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:20:16.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-03T05:20:16.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCQUl1mym50HRAAj0zNEfoRsKq4KJU4JdI5cw== --osd-uuid 5acf6b98-9c70-4fb2-a537-4311095ba06f 2024-06-03T05:20:16.866 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:20:16.861+0000 7fd720f5e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:20:16.878 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:20:16.873+0000 7fd720f5e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:20:16.879 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:20:16.873+0000 7fd720f5e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:20:16.880 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:20:16.873+0000 7fd720f5e7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:20:16.880 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:20:16.873+0000 7fd720f5e7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:20:16.880 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:20:16.873+0000 7fd720f5e7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:20:16.926 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:20:16.921+0000 7fd720f5e7c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-03T05:20:23.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-03T05:20:23.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:20:23.212 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T05:20:23.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T05:20:23.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:20:23.770 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:20:23.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T05:20:23.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:20:23.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:20:23.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:20:23.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:20:23.802 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:20:23.797+0000 7f3f99e167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:20:23.827 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:20:23.821+0000 7f3f99e167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:20:23.828 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:20:23.821+0000 7f3f99e167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:20:24.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T05:20:24.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:20:24.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:20:24.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:20:24.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:20:24.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:20:24.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:20:24.220 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:20:24.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:20:24.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:20:24.394 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:20:24.389+0000 7f3f99e167c0 -1 Falling back to public interface 2024-06-03T05:20:24.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:20:25.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:20:25.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:20:25.669 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:20:25.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:20:25.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:20:25.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:20:26.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:20:27.067 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:20:27.061+0000 7f3f99e167c0 -1 osd.0 0 log_to_monitors true 2024-06-03T05:20:27.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:20:27.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:20:27.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:20:27.081 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:20:27.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:20:27.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:20:27.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:20:28.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:20:28.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:20:28.754 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:20:28.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:20:28.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:20:28.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:20:29.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:20:30.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:20:30.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:20:30.234 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:20:30.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:20:30.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:20:30.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:20:30.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:20:31.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:20:31.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:20:31.662 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:20:31.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:20:31.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:20:31.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:20:32.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:20:33.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:20:33.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:20:33.212 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:20:33.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:20:33.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:20:33.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:20:33.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:20:34.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:20:34.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:20:34.671 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:20:34.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:20:34.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:20:34.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:20:35.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:20:36.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:20:36.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:20:36.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:20:36.234 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:20:36.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:20:36.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:20:36.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:20:37.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:20:37.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:20:37.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:20:37.681 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:20:37.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:20:37.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:20:38.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:20:39.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:20:39.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:20:39.173 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:20:39.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:20:39.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:20:39.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:20:39.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:20:40.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:20:40.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:20:40.587 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:20:40.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:20:40.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:20:40.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:20:41.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:20:42.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:20:42.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:20:42.417 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:20:42.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:20:42.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:20:42.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:20:43.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:20:44.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:20:44.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:20:44.236 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:20:44.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:20:44.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:20:44.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:20:45.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:20:46.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:20:46.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:20:46.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:20:46.182 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:20:46.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:20:46.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:20:47.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:20:48.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:20:48.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:20:48.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:20:48.040 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:20:48.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:20:48.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:20:48.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:20:49.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:20:49.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:20:49.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:20:49.928 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:20:49.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:20:49.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:20:50.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:20:51.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:20:51.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:20:51.583 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:20:51.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:20:51.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:20:51.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:20:51.984 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3126696739,v1:127.0.0.1:6803/3126696739] [v2:127.0.0.1:6804/3126696739,v1:127.0.0.1:6805/3126696739] exists,up 5acf6b98-9c70-4fb2-a537-4311095ba06f 2024-06-03T05:20:51.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:20:51.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:20:51.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:20:51.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1852: test_repair: create_rbd_pool 2024-06-03T05:20:51.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-03T05:20:52.383 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' does not exist 2024-06-03T05:20:52.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-03T05:20:52.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-03T05:20:52.923 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' already exists 2024-06-03T05:20:52.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T05:20:53.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-03T05:20:55.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1853: test_repair: wait_for_clean 2024-06-03T05:20:55.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T05:20:55.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T05:20:55.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T05:20:55.223 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T05:20:55.224 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:20:55.224 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:20:55.224 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:20:55.225 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:20:55.225 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:20:55.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T05:20:55.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T05:20:55.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T05:20:55.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T05:20:55.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T05:20:55.343 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T05:20:55.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-06-03T05:20:55.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T05:20:55.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:20:55.763 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T05:20:55.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836487 2024-06-03T05:20:55.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836487 2024-06-03T05:20:55.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836487' 2024-06-03T05:20:55.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:20:55.928 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836487 2024-06-03T05:20:55.928 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:20:55.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T05:20:55.929 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836487 2024-06-03T05:20:55.929 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:20:55.930 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836487 2024-06-03T05:20:55.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836487 2024-06-03T05:20:55.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-06-03T05:20:55.931 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:20:56.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836485 -lt 21474836487 2024-06-03T05:20:56.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T05:20:57.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T05:20:57.329 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:20:57.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836487 -lt 21474836487 2024-06-03T05:20:57.745 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T05:20:57.746 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:20:57.746 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:20:58.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-03T05:20:58.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T05:20:58.268 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T05:20:58.268 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T05:20:58.268 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T05:20:58.268 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T05:20:58.268 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T05:20:58.268 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T05:20:58.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T05:20:58.651 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T05:20:58.651 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:20:58.651 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:20:59.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-03T05:20:59.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T05:20:59.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T05:20:59.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1854: test_repair: repair 1.0 2024-06-03T05:20:59.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-06-03T05:20:59.188 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-06-03T05:20:59.188 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T05:20:59.188 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T05:20:59.188 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T05:20:59.188 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T05:20:59.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-03T05:20:52.744474+0000 2024-06-03T05:20:59.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-06-03T05:20:59.963 INFO:tasks.workunit.client.0.smithi088.stderr:instructing pg 1.0 on osd.0 to repair 2024-06-03T05:20:59.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-06-03T05:20:52.744474+0000 2024-06-03T05:20:59.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-06-03T05:20:59.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T05:20:52.744474+0000 2024-06-03T05:20:59.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T05:20:59.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T05:20:59.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T05:20:59.980 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T05:20:59.980 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T05:20:59.980 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T05:20:59.981 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T05:20:59.981 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T05:21:00.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T05:20:52.744474+0000 '>' 2024-06-03T05:20:52.744474+0000 2024-06-03T05:21:00.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T05:21:01.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T05:21:01.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T05:21:01.367 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T05:21:01.367 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T05:21:01.367 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T05:21:01.367 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T05:21:01.367 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T05:21:01.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T05:20:52.744474+0000 '>' 2024-06-03T05:20:52.744474+0000 2024-06-03T05:21:01.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T05:21:02.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T05:21:02.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T05:21:02.772 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T05:21:02.772 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T05:21:02.772 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T05:21:02.773 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T05:21:02.773 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T05:21:03.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T05:21:00.049760+0000 '>' 2024-06-03T05:20:52.744474+0000 2024-06-03T05:21:03.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T05:21:03.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: test_repair: kill_daemons td/ceph-helpers KILL osd 2024-06-03T05:21:03.169 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:21:03.169 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:21:03.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:21:03.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:21:03.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:21:03.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:21:03.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: test_repair: TIMEOUT=1 2024-06-03T05:21:03.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: test_repair: repair 1.0 2024-06-03T05:21:03.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-06-03T05:21:03.277 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-06-03T05:21:03.277 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T05:21:03.277 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T05:21:03.277 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T05:21:03.278 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T05:21:03.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-03T05:21:00.049760+0000 2024-06-03T05:21:03.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-06-03T05:21:04.055 INFO:tasks.workunit.client.0.smithi088.stderr:instructing pg 1.0 on osd.0 to repair 2024-06-03T05:21:04.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-06-03T05:21:00.049760+0000 2024-06-03T05:21:04.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-06-03T05:21:04.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T05:21:00.049760+0000 2024-06-03T05:21:04.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T05:21:04.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T05:21:04.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 1 )) 2024-06-03T05:21:04.071 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T05:21:04.071 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T05:21:04.071 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T05:21:04.071 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T05:21:04.072 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T05:21:04.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T05:21:00.049760+0000 '>' 2024-06-03T05:21:00.049760+0000 2024-06-03T05:21:04.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T05:21:05.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T05:21:05.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 1 )) 2024-06-03T05:21:05.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: return 1 2024-06-03T05:21:05.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: test_repair: teardown td/ceph-helpers 2024-06-03T05:21:05.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:21:05.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:21:05.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:21:05.464 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:21:05.464 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:21:05.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:21:05.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:21:05.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:21:05.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:21:05.575 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:21:05.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:21:05.576 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:21:05.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:21:05.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:21:05.577 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:21:05.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:21:05.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:21:05.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:21:05.579 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:21:05.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:21:05.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:21:05.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:21:05.590 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:21:05.590 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:21:05.590 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:21:05.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:21:05.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:21:05.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:21:05.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:21:05.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_run_in_background td/ceph-helpers 2024-06-03T05:21:05.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: test_run_in_background: local pids 2024-06-03T05:21:05.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: test_run_in_background: run_in_background pids sleep 1 2024-06-03T05:21:05.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T05:21:05.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T05:21:05.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 131802"' 2024-06-03T05:21:05.594 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 131802' 2024-06-03T05:21:05.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: test_run_in_background: run_in_background pids sleep 1 2024-06-03T05:21:05.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T05:21:05.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T05:21:05.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 131803"' 2024-06-03T05:21:05.594 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 131803' 2024-06-03T05:21:05.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/131805: /' 2024-06-03T05:21:05.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sleep 1 2024-06-03T05:21:05.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sleep 1 2024-06-03T05:21:05.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/131808: /' 2024-06-03T05:21:05.594 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: test_run_in_background: echo 131802 131803 2024-06-03T05:21:05.595 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: test_run_in_background: wc -w 2024-06-03T05:21:05.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: test_run_in_background: test 2 = 2 2024-06-03T05:21:05.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: test_run_in_background: wait 131802 131803 2024-06-03T05:21:06.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T05:21:06.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T05:21:06.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:21:06.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_run_mon td/ceph-helpers 2024-06-03T05:21:06.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:495: test_run_mon: local dir=td/ceph-helpers 2024-06-03T05:21:06.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:497: test_run_mon: setup td/ceph-helpers 2024-06-03T05:21:06.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-03T05:21:06.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-03T05:21:06.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:21:06.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:21:06.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:21:06.596 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:21:06.596 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:21:06.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:21:06.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:21:06.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:21:06.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:21:06.598 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:21:06.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:21:06.600 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:21:06.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:21:06.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:21:06.601 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:21:06.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:21:06.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:21:06.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:21:06.603 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:21:06.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:21:06.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:21:06.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:21:06.607 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:21:06.607 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:21:06.608 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:21:06.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:21:06.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:21:06.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:21:06.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-03T05:21:06.610 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:21:06.610 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:21:06.610 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:21:06.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70897 2024-06-03T05:21:06.612 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:21:06.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:21:06.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:21:06.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-03T05:21:06.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:499: test_run_mon: run_mon td/ceph-helpers a 2024-06-03T05:21:06.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-03T05:21:06.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:21:06.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:21:06.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:21:06.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-03T05:21:06.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-06-03T05:21:06.999 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:21:06.999 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:21:06.999 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:21:07.000 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:21:07.000 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:21:07.000 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:21:07.000 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:21:07.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-03T05:21:07.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:21:07.287 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:21:07.287 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:21:07.287 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:21:07.287 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:21:07.288 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:21:07.288 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:21:07.288 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:21:07.289 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:21:07.289 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:21:07.289 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:21:07.289 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:21:07.289 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:21:07.289 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:21:07.290 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get fsid 2024-06-03T05:21:07.403 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:21:07.403 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:21:07.403 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:21:07.403 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:21:07.403 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:21:07.404 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:21:07.404 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:21:07.404 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:21:07.405 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:21:07.405 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:21:07.405 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:21:07.405 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:21:07.405 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:21:07.405 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get mon_host 2024-06-03T05:21:07.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:500: test_run_mon: grep mon.a 2024-06-03T05:21:07.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:500: test_run_mon: ceph mon dump 2024-06-03T05:21:08.346 INFO:tasks.workunit.client.0.smithi088.stderr:dumped monmap epoch 1 2024-06-03T05:21:08.361 INFO:tasks.workunit.client.0.smithi088.stdout:0: v2:127.0.0.1:7109/0 mon.a 2024-06-03T05:21:08.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:501: test_run_mon: kill_daemons td/ceph-helpers 2024-06-03T05:21:08.363 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:21:08.363 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:21:08.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:21:08.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:21:08.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:21:08.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:21:08.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:503: test_run_mon: run_mon td/ceph-helpers a --osd_pool_default_size=3 2024-06-03T05:21:08.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-03T05:21:08.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:21:08.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:21:08.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:21:08.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-03T05:21:08.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=3 2024-06-03T05:21:08.702 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:21:08.697+0000 7fd8ff8b3c80 -1 'td/ceph-helpers/a' already exists and is not empty: monitor may already exist 2024-06-03T05:21:08.704 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:21:08.704 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:21:08.704 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:21:08.705 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:21:08.705 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:21:08.705 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:21:08.705 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:21:08.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-06-03T05:21:08.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:21:08.929 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:21:08.929 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:21:08.929 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:21:08.929 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:21:08.929 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:21:08.930 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:21:08.930 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:21:08.930 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:21:08.930 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:21:08.930 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:21:08.931 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:21:08.931 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:21:08.931 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:21:08.931 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get fsid 2024-06-03T05:21:09.047 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:21:09.047 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:21:09.047 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:21:09.048 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:21:09.048 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:21:09.049 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:21:09.049 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:21:09.049 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:21:09.049 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:21:09.049 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:21:09.049 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:21:09.049 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:21:09.050 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:21:09.050 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get mon_host 2024-06-03T05:21:09.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:504: test_run_mon: run_osd td/ceph-helpers 0 2024-06-03T05:21:09.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:21:09.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:21:09.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T05:21:09.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:21:09.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-03T05:21:09.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:21:09.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:21:09.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:21:09.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:21:09.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-03T05:21:09.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-03T05:21:09.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:21:09.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:21:09.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:21:09.163 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:21:09.163 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:21:09.163 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:21:09.163 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:21:09.163 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:21:09.163 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:21:09.163 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:21:09.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:21:09.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:21:09.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:21:09.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:21:09.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:21:09.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:21:09.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:21:09.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:21:09.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:21:09.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:21:09.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:21:09.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:21:09.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-03T05:21:09.165 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:21:09.165 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 302ed505-f6d7-4404-97a3-d98107d11047 2024-06-03T05:21:09.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=302ed505-f6d7-4404-97a3-d98107d11047 2024-06-03T05:21:09.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 302ed505-f6d7-4404-97a3-d98107d11047' 2024-06-03T05:21:09.166 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:21:09.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDFUl1mUPX6ChAA4QZ8/LYE8quVsQqSEsNiZg== 2024-06-03T05:21:09.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDFUl1mUPX6ChAA4QZ8/LYE8quVsQqSEsNiZg=="}' 2024-06-03T05:21:09.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 302ed505-f6d7-4404-97a3-d98107d11047 -i td/ceph-helpers/0/new.json 2024-06-03T05:21:09.493 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:21:09.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-03T05:21:09.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDFUl1mUPX6ChAA4QZ8/LYE8quVsQqSEsNiZg== --osd-uuid 302ed505-f6d7-4404-97a3-d98107d11047 2024-06-03T05:21:09.538 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:21:09.533+0000 7f32048ac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:21:09.569 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:21:09.565+0000 7f32048ac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:21:09.570 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:21:09.565+0000 7f32048ac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:21:09.571 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:21:09.565+0000 7f32048ac7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:21:09.571 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:21:09.565+0000 7f32048ac7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:21:09.571 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:21:09.565+0000 7f32048ac7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:21:09.609 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:21:09.605+0000 7f32048ac7c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-03T05:21:15.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-03T05:21:15.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:21:15.867 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T05:21:15.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T05:21:15.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:21:16.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T05:21:16.119 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:21:16.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:21:16.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:21:16.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:21:16.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:21:16.151 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:21:16.145+0000 7f595c34e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:21:16.153 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:21:16.149+0000 7f595c34e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:21:16.155 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:21:16.149+0000 7f595c34e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:21:16.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T05:21:16.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:21:16.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:21:16.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:21:16.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:21:16.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:21:16.346 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:21:16.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:21:16.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:21:16.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:21:16.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:21:16.700 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:21:16.693+0000 7f595c34e7c0 -1 Falling back to public interface 2024-06-03T05:21:17.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:21:17.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:21:17.560 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:21:17.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:21:17.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:21:17.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:21:17.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:21:18.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:21:18.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:21:18.792 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:21:18.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:21:18.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:21:18.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:21:19.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:21:19.363 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:21:19.357+0000 7f595c34e7c0 -1 osd.0 0 log_to_monitors true 2024-06-03T05:21:20.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:21:20.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:21:20.021 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:21:20.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:21:20.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:21:20.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:21:20.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:21:21.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:21:21.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:21:21.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:21:21.256 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:21:21.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:21:21.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:21:21.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:21:22.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:21:22.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:21:22.493 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:21:22.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:21:22.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:21:22.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:21:22.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:21:23.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:21:23.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:21:23.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:21:23.732 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:21:23.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:21:23.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:21:23.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:21:24.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:21:24.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:21:24.970 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:21:24.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:21:24.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:21:24.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:21:25.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:21:26.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:21:26.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:21:26.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:21:26.212 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:21:26.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:21:26.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:21:26.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:21:27.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:21:27.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:21:27.447 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:21:27.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:21:27.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:21:27.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:21:27.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:21:28.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:21:28.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:21:28.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:21:28.675 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:21:28.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:21:28.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:21:28.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:21:29.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:21:29.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:21:29.906 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:21:29.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:21:29.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:21:29.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:21:30.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:21:31.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:21:31.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:21:31.138 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:21:31.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:21:31.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:21:31.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:21:31.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:21:32.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:21:32.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:21:32.367 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:21:32.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:21:32.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:21:32.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:21:32.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:21:33.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:21:33.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:21:33.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:21:33.604 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:21:33.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:21:33.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:21:33.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:21:34.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:21:34.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:21:34.840 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:21:34.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:21:34.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:21:34.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:21:35.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:21:36.079 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:21:36.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:21:36.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:21:36.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:21:36.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:21:36.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:21:36.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:21:37.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:21:37.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:21:37.318 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:21:37.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:21:37.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:21:37.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:21:37.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:21:38.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:21:38.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:21:38.558 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-06-03T05:21:38.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T05:21:38.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:21:38.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:21:38.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:21:39.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:21:39.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:21:39.796 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-06-03T05:21:39.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-03T05:21:39.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:21:39.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:21:40.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:21:41.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:21:41.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:21:41.034 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-06-03T05:21:41.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-03T05:21:41.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:21:41.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:21:41.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:21:42.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:21:42.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:21:42.272 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-06-03T05:21:42.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-03T05:21:42.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:21:42.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:21:42.662 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3333995290,v1:127.0.0.1:6801/3333995290] [v2:127.0.0.1:6802/3333995290,v1:127.0.0.1:6803/3333995290] exists,up 302ed505-f6d7-4404-97a3-d98107d11047 2024-06-03T05:21:42.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:21:42.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:21:42.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:21:42.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:505: test_run_mon: run_osd td/ceph-helpers 1 2024-06-03T05:21:42.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:21:42.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:21:42.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-03T05:21:42.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:21:42.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-06-03T05:21:42.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:21:42.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:21:42.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:21:42.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:21:42.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-06-03T05:21:42.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-06-03T05:21:42.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:21:42.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:21:42.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:21:42.665 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:21:42.665 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:21:42.666 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:21:42.666 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:21:42.666 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:21:42.666 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:21:42.666 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:21:42.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:21:42.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:21:42.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:21:42.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:21:42.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:21:42.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:21:42.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:21:42.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:21:42.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:21:42.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:21:42.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:21:42.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:21:42.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-06-03T05:21:42.668 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:21:42.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b6f33838-eaad-48a1-b5fa-433753057ce1 2024-06-03T05:21:42.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 b6f33838-eaad-48a1-b5fa-433753057ce1' 2024-06-03T05:21:42.669 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 b6f33838-eaad-48a1-b5fa-433753057ce1 2024-06-03T05:21:42.669 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:21:42.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDmUl1m5jkPKRAAZV+Xc1P9ueqPMXKfDkSSxg== 2024-06-03T05:21:42.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDmUl1m5jkPKRAAZV+Xc1P9ueqPMXKfDkSSxg=="}' 2024-06-03T05:21:42.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b6f33838-eaad-48a1-b5fa-433753057ce1 -i td/ceph-helpers/1/new.json 2024-06-03T05:21:43.078 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:21:43.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-06-03T05:21:43.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDmUl1m5jkPKRAAZV+Xc1P9ueqPMXKfDkSSxg== --osd-uuid b6f33838-eaad-48a1-b5fa-433753057ce1 2024-06-03T05:21:43.130 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:21:43.124+0000 7f79cc73c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:21:43.164 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:21:43.156+0000 7f79cc73c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:21:43.165 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:21:43.160+0000 7f79cc73c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:21:43.166 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:21:43.160+0000 7f79cc73c7c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-03T05:21:43.166 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:21:43.160+0000 7f79cc73c7c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-03T05:21:43.166 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:21:43.160+0000 7f79cc73c7c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-03T05:21:43.203 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:21:43.196+0000 7f79cc73c7c0 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-06-03T05:21:49.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-06-03T05:21:49.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:21:49.392 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-06-03T05:21:49.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-03T05:21:49.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:21:49.640 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-06-03T05:21:49.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-03T05:21:49.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:21:49.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:21:49.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:21:49.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:21:49.672 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:21:49.664+0000 7fb0913af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:21:49.675 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:21:49.668+0000 7fb0913af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:21:49.676 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:21:49.668+0000 7fb0913af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:21:49.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-03T05:21:49.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:21:49.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T05:21:49.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:21:49.869 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:21:49.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:21:49.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:21:49.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:21:49.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:21:49.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:21:50.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:21:50.232 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:21:50.224+0000 7fb0913af7c0 -1 Falling back to public interface 2024-06-03T05:21:51.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:21:51.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:21:51.103 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:21:51.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:21:51.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:21:51.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:21:51.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:21:52.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:21:52.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:21:52.341 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:21:52.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:21:52.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:21:52.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:21:52.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:21:52.581 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:21:52.580+0000 7fb0913af7c0 -1 osd.1 0 log_to_monitors true 2024-06-03T05:21:53.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:21:53.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:21:53.580 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:21:53.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:21:53.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:21:53.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:21:53.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:21:54.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:21:54.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:21:54.877 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:21:54.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:21:54.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:21:54.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:21:55.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:21:56.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:21:56.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:21:56.107 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:21:56.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:21:56.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:21:56.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:21:56.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:21:57.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:21:57.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:21:57.334 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:21:57.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:21:57.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:21:57.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:21:57.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:21:58.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:21:58.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:21:58.569 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:21:58.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:21:58.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:21:58.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:21:58.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:21:59.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:21:59.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:21:59.810 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:21:59.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:21:59.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:21:59.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:22:00.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:22:01.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:22:01.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:22:01.049 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:22:01.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:22:01.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:22:01.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:22:01.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:22:02.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:22:02.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:22:02.285 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:22:02.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:22:02.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:22:02.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:22:02.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:22:03.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:22:03.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:22:03.524 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:22:03.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:22:03.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:22:03.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:22:03.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:22:04.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:22:04.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:22:04.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:22:04.760 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:22:04.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:22:04.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:22:04.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:22:05.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:22:05.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:22:05.995 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:22:05.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:22:05.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:22:05.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:22:06.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:22:07.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:22:07.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:22:07.232 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:22:07.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:22:07.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:22:07.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:22:07.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:22:08.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:22:08.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:22:08.460 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:22:08.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:22:08.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:22:08.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:22:08.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:22:09.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:22:09.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:22:09.697 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:22:09.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:22:09.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:22:09.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:22:09.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:22:10.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:22:10.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:22:10.931 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:22:10.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:22:10.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:22:10.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:22:11.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:22:12.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:22:12.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:22:12.162 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-06-03T05:22:12.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T05:22:12.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:22:12.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:22:12.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:22:13.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:22:13.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:22:13.401 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-06-03T05:22:13.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-03T05:22:13.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:22:13.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:22:13.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:22:14.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:22:14.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:22:14.637 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-06-03T05:22:14.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-03T05:22:14.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:22:14.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:22:14.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:22:15.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:22:15.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:22:15.876 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-06-03T05:22:15.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-03T05:22:15.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:22:15.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:22:16.138 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/2513871338,v1:127.0.0.1:6809/2513871338] [v2:127.0.0.1:6810/2513871338,v1:127.0.0.1:6811/2513871338] exists,up b6f33838-eaad-48a1-b5fa-433753057ce1 2024-06-03T05:22:16.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:22:16.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:22:16.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:22:16.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:506: test_run_mon: run_osd td/ceph-helpers 2 2024-06-03T05:22:16.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:22:16.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:22:16.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-03T05:22:16.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:22:16.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/2 2024-06-03T05:22:16.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:22:16.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:22:16.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:22:16.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:22:16.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2024-06-03T05:22:16.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2024-06-03T05:22:16.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:22:16.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:22:16.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:22:16.145 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:22:16.145 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:22:16.145 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:22:16.145 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:22:16.145 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:22:16.145 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:22:16.145 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:22:16.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:22:16.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:22:16.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:22:16.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:22:16.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:22:16.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:22:16.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:22:16.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:22:16.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:22:16.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:22:16.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:22:16.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:22:16.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/2 2024-06-03T05:22:16.146 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:22:16.146 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 31c550e4-cec3-4ed9-9109-318c10b13cff 2024-06-03T05:22:16.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=31c550e4-cec3-4ed9-9109-318c10b13cff 2024-06-03T05:22:16.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 31c550e4-cec3-4ed9-9109-318c10b13cff' 2024-06-03T05:22:16.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:22:16.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAIU11mN83ICRAAAosSp5SzkE3hl+ku6Xc5vA== 2024-06-03T05:22:16.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAIU11mN83ICRAAAosSp5SzkE3hl+ku6Xc5vA=="}' 2024-06-03T05:22:16.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 31c550e4-cec3-4ed9-9109-318c10b13cff -i td/ceph-helpers/2/new.json 2024-06-03T05:22:16.529 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:22:16.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/2/new.json 2024-06-03T05:22:16.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAIU11mN83ICRAAAosSp5SzkE3hl+ku6Xc5vA== --osd-uuid 31c550e4-cec3-4ed9-9109-318c10b13cff 2024-06-03T05:22:16.582 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:22:16.580+0000 7f17534ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:22:16.623 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:22:16.620+0000 7f17534ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:22:16.625 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:22:16.624+0000 7f17534ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:22:16.626 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:22:16.624+0000 7f17534ab7c0 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-06-03T05:22:16.626 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:22:16.624+0000 7f17534ab7c0 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-06-03T05:22:16.626 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:22:16.624+0000 7f17534ab7c0 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-06-03T05:22:16.663 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:22:16.660+0000 7f17534ab7c0 -1 bluestore(td/ceph-helpers/2) _read_fsid unparsable uuid 2024-06-03T05:22:23.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/2/keyring 2024-06-03T05:22:23.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:22:23.025 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-06-03T05:22:23.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-03T05:22:23.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:22:23.266 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-06-03T05:22:23.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-03T05:22:23.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:22:23.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:22:23.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:22:23.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:22:23.299 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:22:23.296+0000 7f95192ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:22:23.300 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:22:23.296+0000 7f95192ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:22:23.302 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:22:23.300+0000 7f95192ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:22:23.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-03T05:22:23.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:22:23.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T05:22:23.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:22:23.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:22:23.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:22:23.496 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:22:23.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:22:23.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:22:23.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:22:23.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:22:23.859 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:22:23.856+0000 7f95192ed7c0 -1 Falling back to public interface 2024-06-03T05:22:24.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:22:24.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:22:24.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:22:24.720 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:22:24.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:22:24.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:22:24.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:22:25.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:22:25.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:22:25.956 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:22:25.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:22:25.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:22:25.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:22:26.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:22:26.488 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:22:26.484+0000 7f95192ed7c0 -1 osd.2 0 log_to_monitors true 2024-06-03T05:22:27.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:22:27.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:22:27.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:22:27.194 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:22:27.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:22:27.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:22:27.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:22:28.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:22:28.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:22:28.624 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:22:28.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:22:28.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:22:28.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:22:28.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:22:29.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:22:29.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:22:29.861 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:22:29.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:22:29.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:22:29.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:22:30.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:22:31.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:22:31.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:22:31.101 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:22:31.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:22:31.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:22:31.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:22:31.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:22:32.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:22:32.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:22:32.337 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:22:32.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:22:32.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:22:32.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:22:32.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:22:33.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:22:33.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:22:33.566 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:22:33.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:22:33.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:22:33.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:22:33.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:22:34.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:22:34.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:22:34.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:22:34.802 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:22:34.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:22:34.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:22:35.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:22:36.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:22:36.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:22:36.041 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:22:36.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:22:36.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:22:36.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:22:36.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:22:37.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:22:37.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:22:37.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:22:37.268 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:22:37.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:22:37.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:22:37.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:22:38.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:22:38.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:22:38.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:22:38.505 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:22:38.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:22:38.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:22:38.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:22:39.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:22:39.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:22:39.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:22:39.741 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:22:39.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:22:39.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:22:39.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:22:40.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:22:40.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:22:40.976 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:22:40.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:22:40.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:22:40.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:22:41.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:22:42.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:22:42.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:22:42.203 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:22:42.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:22:42.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:22:42.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:22:42.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:22:43.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:22:43.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:22:43.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:22:43.441 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:22:43.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:22:43.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:22:43.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:22:44.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:22:44.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:22:44.677 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:22:44.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:22:44.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:22:44.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:22:44.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:22:45.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:22:45.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:22:45.916 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-06-03T05:22:45.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T05:22:45.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:22:45.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:22:46.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:22:47.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:22:47.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:22:47.153 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-06-03T05:22:47.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-03T05:22:47.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:22:47.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:22:47.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:22:48.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:22:48.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:22:48.383 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-06-03T05:22:48.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-03T05:22:48.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:22:48.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:22:48.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:22:49.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:22:49.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:22:49.608 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-06-03T05:22:49.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-03T05:22:49.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:22:49.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:22:49.837 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/2119564771,v1:127.0.0.1:6817/2119564771] [v2:127.0.0.1:6818/2119564771,v1:127.0.0.1:6819/2119564771] exists,up 31c550e4-cec3-4ed9-9109-318c10b13cff 2024-06-03T05:22:49.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:22:49.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:22:49.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:22:49.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:507: test_run_mon: create_rbd_pool 2024-06-03T05:22:49.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-03T05:22:50.076 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' does not exist 2024-06-03T05:22:50.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-03T05:22:50.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-03T05:22:50.468 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' already exists 2024-06-03T05:22:50.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T05:22:51.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-03T05:22:53.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:508: test_run_mon: ceph osd dump 2024-06-03T05:22:53.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:508: test_run_mon: grep 'pool 1 '\''rbd'\''' 2024-06-03T05:22:53.880 INFO:tasks.workunit.client.0.smithi088.stdout:pool 1 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 4 pgp_num 4 autoscale_mode off last_change 23 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 1.50 2024-06-03T05:22:53.880 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:510: test_run_mon: get_asok_path mon.a 2024-06-03T05:22:53.880 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:22:53.881 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:22:53.881 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:22:53.882 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:22:53.882 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:22:53.882 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:22:53.882 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:510: test_run_mon: CEPH_ARGS= 2024-06-03T05:22:53.882 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:510: test_run_mon: ceph --format=json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get osd_pool_default_size 2024-06-03T05:22:53.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:510: test_run_mon: local 'size={"osd_pool_default_size":"3"}' 2024-06-03T05:22:53.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:511: test_run_mon: test '{"osd_pool_default_size":"3"}' = '{"osd_pool_default_size":"3"}' 2024-06-03T05:22:53.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:513: test_run_mon: CEPH_ARGS= 2024-06-03T05:22:53.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:513: test_run_mon: ceph status 2024-06-03T05:22:54.106 INFO:tasks.workunit.client.0.smithi088.stderr:unable to get monitor info from DNS SRV with service name: ceph-mon 2024-06-03T05:22:54.106 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:22:54.104+0000 7f0254063640 -1 failed for service _ceph-mon._tcp 2024-06-03T05:22:54.106 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:22:54.104+0000 7f0254063640 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-06-03T05:22:54.106 INFO:tasks.workunit.client.0.smithi088.stderr:[errno 2] RADOS object not found (error connecting to the cluster) 2024-06-03T05:22:54.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:514: test_run_mon: CEPH_ARGS= 2024-06-03T05:22:54.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:514: test_run_mon: ceph --conf td/ceph-helpers/ceph.conf status 2024-06-03T05:22:54.210 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:22:54.208+0000 7f44a22f6640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-06-03T05:22:54.211 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:22:54.208+0000 7f44a22f6640 -1 AuthRegistry(0x7f449c068990) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-06-03T05:22:54.211 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:22:54.208+0000 7f44a22f6640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-06-03T05:22:54.211 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:22:54.208+0000 7f44a22f6640 -1 AuthRegistry(0x7f44a22f4f50) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-06-03T05:22:54.213 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:22:54.208+0000 7f44a22f6640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-06-03T05:22:54.213 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:22:54.208+0000 7f44a22f6640 -1 AuthRegistry(0x7f449c065b40) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-06-03T05:22:54.213 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:22:54.208+0000 7f44a22f6640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-06-03T05:22:54.213 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:22:54.208+0000 7f44a22f6640 -1 AuthRegistry(0x7f449c1098f0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-06-03T05:22:54.213 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:22:54.208+0000 7f44a22f6640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-06-03T05:22:54.214 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:22:54.208+0000 7f44a22f6640 -1 AuthRegistry(0x7f449c065b40) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-06-03T05:22:54.316 INFO:tasks.workunit.client.0.smithi088.stdout: cluster: 2024-06-03T05:22:54.316 INFO:tasks.workunit.client.0.smithi088.stdout: id: 213fb4bd-379a-40f7-a383-f6c59451eb9c 2024-06-03T05:22:54.316 INFO:tasks.workunit.client.0.smithi088.stdout: health: HEALTH_OK 2024-06-03T05:22:54.316 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-06-03T05:22:54.316 INFO:tasks.workunit.client.0.smithi088.stdout: services: 2024-06-03T05:22:54.316 INFO:tasks.workunit.client.0.smithi088.stdout: mon: 1 daemons, quorum a (age 105s) 2024-06-03T05:22:54.316 INFO:tasks.workunit.client.0.smithi088.stdout: mgr: no daemons active 2024-06-03T05:22:54.316 INFO:tasks.workunit.client.0.smithi088.stdout: osd: 3 osds: 3 up (since 5s), 3 in (since 37s) 2024-06-03T05:22:54.316 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-06-03T05:22:54.316 INFO:tasks.workunit.client.0.smithi088.stdout: data: 2024-06-03T05:22:54.316 INFO:tasks.workunit.client.0.smithi088.stdout: pools: 0 pools, 0 pgs 2024-06-03T05:22:54.316 INFO:tasks.workunit.client.0.smithi088.stdout: objects: 0 objects, 0 B 2024-06-03T05:22:54.317 INFO:tasks.workunit.client.0.smithi088.stdout: usage: 0 B used, 0 B / 0 B avail 2024-06-03T05:22:54.317 INFO:tasks.workunit.client.0.smithi088.stdout: pgs: 2024-06-03T05:22:54.317 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-06-03T05:22:54.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:516: test_run_mon: kill_daemons td/ceph-helpers 2024-06-03T05:22:54.331 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:22:54.331 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:22:54.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:22:54.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:22:54.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:22:59.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:22:59.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:518: test_run_mon: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:22:59.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-03T05:22:59.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:22:59.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:22:59.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:22:59.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-03T05:22:59.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:22:59.677 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:22:59.672+0000 7fc0c7409c80 -1 'td/ceph-helpers/a' already exists and is not empty: monitor may already exist 2024-06-03T05:22:59.680 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:22:59.680 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:22:59.680 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:22:59.681 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:22:59.681 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:22:59.681 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:22:59.682 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:22:59.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:22:59.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:22:59.922 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:22:59.922 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:22:59.922 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:22:59.922 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:22:59.922 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:22:59.923 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:22:59.923 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:22:59.923 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:22:59.923 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:22:59.924 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:22:59.924 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:22:59.924 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:22:59.924 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:22:59.924 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get fsid 2024-06-03T05:23:00.044 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:23:00.044 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:23:00.044 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:23:00.044 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:23:00.046 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:23:00.046 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:23:00.046 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:23:00.046 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:23:00.046 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:23:00.046 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:23:00.046 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:23:00.046 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:23:00.046 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:23:00.046 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get mon_host 2024-06-03T05:23:00.154 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: get_asok_path mon.a 2024-06-03T05:23:00.154 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:23:00.154 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:23:00.154 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:23:00.154 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:23:00.154 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:23:00.154 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:23:00.155 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: CEPH_ARGS= 2024-06-03T05:23:00.155 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: ceph --format=json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get osd_pool_default_size 2024-06-03T05:23:00.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: local 'size={"osd_pool_default_size":"1"}' 2024-06-03T05:23:00.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:521: test_run_mon: test '{"osd_pool_default_size":"1"}' = '{"osd_pool_default_size":"1"}' 2024-06-03T05:23:00.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:522: test_run_mon: kill_daemons td/ceph-helpers 2024-06-03T05:23:00.263 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:23:00.263 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:23:00.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:23:00.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:23:00.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:23:00.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:23:00.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:524: test_run_mon: CEPH_ARGS='--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd_pool_default_size=2' 2024-06-03T05:23:00.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:524: test_run_mon: run_mon td/ceph-helpers a 2024-06-03T05:23:00.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-03T05:23:00.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:23:00.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:23:00.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:23:00.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-03T05:23:00.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-06-03T05:23:00.400 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:23:00.396+0000 7ff0b4fe3c80 -1 'td/ceph-helpers/a' already exists and is not empty: monitor may already exist 2024-06-03T05:23:00.402 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:23:00.402 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:23:00.402 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:23:00.403 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:23:00.403 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:23:00.403 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:23:00.403 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:23:00.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-03T05:23:00.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:23:00.652 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:23:00.652 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:23:00.652 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:23:00.653 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:23:00.654 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:23:00.654 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:23:00.654 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:23:00.654 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:23:00.654 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:23:00.654 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:23:00.654 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:23:00.654 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:23:00.655 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:23:00.655 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get fsid 2024-06-03T05:23:00.767 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:23:00.767 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:23:00.767 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:23:00.767 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:23:00.767 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:23:00.768 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:23:00.768 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:23:00.768 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:23:00.768 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:23:00.768 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:23:00.769 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:23:00.769 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:23:00.769 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:23:00.769 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get mon_host 2024-06-03T05:23:00.886 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: test_run_mon: get_asok_path mon.a 2024-06-03T05:23:00.886 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:23:00.886 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:23:00.886 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:23:00.886 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:23:00.887 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:23:00.887 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:23:00.887 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: test_run_mon: CEPH_ARGS= 2024-06-03T05:23:00.887 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: test_run_mon: ceph --format=json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get osd_pool_default_size 2024-06-03T05:23:01.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: test_run_mon: local 'size={"osd_pool_default_size":"2"}' 2024-06-03T05:23:01.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: test_run_mon: test '{"osd_pool_default_size":"2"}' = '{"osd_pool_default_size":"2"}' 2024-06-03T05:23:01.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:529: test_run_mon: kill_daemons td/ceph-helpers 2024-06-03T05:23:01.001 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:23:01.001 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:23:01.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:23:01.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:23:01.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:23:01.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:23:01.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:531: test_run_mon: teardown td/ceph-helpers 2024-06-03T05:23:01.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:23:01.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:23:01.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:23:01.111 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:23:01.111 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:23:01.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:23:01.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:23:01.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:23:01.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:23:01.117 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:23:01.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:23:01.118 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:23:01.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:23:01.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:23:01.119 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:23:01.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:23:01.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:23:01.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:23:01.121 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:23:01.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:23:01.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:23:01.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:23:01.135 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:23:01.135 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:23:01.135 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:23:01.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:23:01.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:23:01.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:23:01.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:23:01.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_run_osd td/ceph-helpers 2024-06-03T05:23:01.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:745: test_run_osd: local dir=td/ceph-helpers 2024-06-03T05:23:01.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:747: test_run_osd: setup td/ceph-helpers 2024-06-03T05:23:01.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-03T05:23:01.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-03T05:23:01.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:23:01.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:23:01.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:23:01.138 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:23:01.138 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:23:01.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:23:01.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:23:01.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:23:01.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:23:01.139 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:23:01.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:23:01.140 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:23:01.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:23:01.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:23:01.141 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:23:01.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:23:01.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:23:01.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:23:01.142 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:23:01.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:23:01.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:23:01.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:23:01.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:23:01.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:23:01.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:23:01.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:23:01.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:23:01.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:23:01.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-03T05:23:01.148 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:23:01.148 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:23:01.148 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:23:01.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70897 2024-06-03T05:23:01.149 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:23:01.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:23:01.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:23:01.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-03T05:23:01.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:749: test_run_osd: run_mon td/ceph-helpers a 2024-06-03T05:23:01.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-03T05:23:01.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:23:01.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:23:01.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:23:01.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-03T05:23:01.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-06-03T05:23:01.554 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:23:01.554 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:23:01.554 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:23:01.554 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:23:01.555 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:23:01.555 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:23:01.555 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:23:01.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-03T05:23:01.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:23:01.766 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:23:01.766 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:23:01.766 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:23:01.766 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:23:01.767 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:23:01.767 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:23:01.767 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:23:01.768 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:23:01.768 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:23:01.768 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:23:01.768 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:23:01.768 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:23:01.768 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:23:01.768 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get fsid 2024-06-03T05:23:01.879 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:23:01.879 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:23:01.879 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:23:01.879 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:23:01.880 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:23:01.880 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:23:01.881 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:23:01.881 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:23:01.881 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:23:01.881 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:23:01.881 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:23:01.881 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:23:01.881 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:23:01.881 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get mon_host 2024-06-03T05:23:01.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:750: test_run_osd: run_mgr td/ceph-helpers x 2024-06-03T05:23:01.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-03T05:23:01.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T05:23:01.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T05:23:01.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T05:23:01.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-03T05:23:01.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T05:23:02.880 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T05:23:02.881 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:23:02.881 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:23:02.881 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:23:02.881 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:23:02.881 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:23:02.881 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:23:02.881 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:23:02.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:23:02.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:752: test_run_osd: run_osd td/ceph-helpers 0 2024-06-03T05:23:02.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:23:02.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:23:02.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T05:23:02.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:23:02.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-03T05:23:02.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:23:02.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:23:02.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:23:02.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:23:02.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-03T05:23:02.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-03T05:23:02.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:23:02.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:23:02.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:23:02.920 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:23:02.920 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:23:02.920 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:23:02.920 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:23:02.920 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:23:02.920 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:23:02.920 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:23:02.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:23:02.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:23:02.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:23:02.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:23:02.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:23:02.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:23:02.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:23:02.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:23:02.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:23:02.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:23:02.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:23:02.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:23:02.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-03T05:23:02.921 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:23:02.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e83d7eeb-41c0-4dc3-a9f9-9349b45d3e5c 2024-06-03T05:23:02.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e83d7eeb-41c0-4dc3-a9f9-9349b45d3e5c' 2024-06-03T05:23:02.922 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 e83d7eeb-41c0-4dc3-a9f9-9349b45d3e5c 2024-06-03T05:23:02.922 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:23:02.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA2U11mqEPSNxAAMnomMRiv/C+WJxMnb691Eg== 2024-06-03T05:23:02.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA2U11mqEPSNxAAMnomMRiv/C+WJxMnb691Eg=="}' 2024-06-03T05:23:02.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e83d7eeb-41c0-4dc3-a9f9-9349b45d3e5c -i td/ceph-helpers/0/new.json 2024-06-03T05:23:03.253 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:23:03.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-03T05:23:03.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA2U11mqEPSNxAAMnomMRiv/C+WJxMnb691Eg== --osd-uuid e83d7eeb-41c0-4dc3-a9f9-9349b45d3e5c 2024-06-03T05:23:03.303 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:23:03.300+0000 7f7214f057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:23:03.321 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:23:03.320+0000 7f7214f057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:23:03.322 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:23:03.320+0000 7f7214f057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:23:03.323 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:23:03.320+0000 7f7214f057c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:23:03.323 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:23:03.320+0000 7f7214f057c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:23:03.323 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:23:03.320+0000 7f7214f057c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:23:03.361 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:23:03.360+0000 7f7214f057c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-03T05:23:09.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-03T05:23:09.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:23:09.559 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T05:23:09.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T05:23:09.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:23:10.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T05:23:10.105 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:23:10.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:23:10.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:23:10.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:23:10.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:23:10.131 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:23:10.128+0000 7fe87dffa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:23:10.133 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:23:10.128+0000 7fe87dffa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:23:10.134 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:23:10.132+0000 7fe87dffa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:23:10.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T05:23:10.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:23:10.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:23:10.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:23:10.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:23:10.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:23:10.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:23:10.524 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:23:10.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:23:10.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:23:10.678 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:23:10.676+0000 7fe87dffa7c0 -1 Falling back to public interface 2024-06-03T05:23:10.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:23:11.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:23:11.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:23:11.967 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:23:11.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:23:11.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:23:11.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:23:12.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:23:13.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:23:13.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:23:13.381 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:23:13.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:23:13.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:23:13.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:23:13.386 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:23:13.384+0000 7fe87dffa7c0 -1 osd.0 0 log_to_monitors true 2024-06-03T05:23:13.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:23:14.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:23:14.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:23:14.850 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:23:14.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:23:14.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:23:14.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:23:15.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:23:16.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:23:16.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:23:16.297 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:23:16.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:23:16.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:23:16.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:23:16.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:23:17.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:23:17.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:23:17.741 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:23:17.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:23:17.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:23:17.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:23:18.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:23:19.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:23:19.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:23:19.283 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:23:19.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:23:19.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:23:19.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:23:19.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:23:20.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:23:20.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:23:20.705 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:23:20.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:23:20.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:23:20.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:23:21.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:23:22.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:23:22.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:23:22.215 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:23:22.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:23:22.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:23:22.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:23:22.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:23:23.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:23:23.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:23:23.742 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:23:23.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:23:23.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:23:23.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:23:24.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:23:25.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:23:25.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:23:25.373 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:23:25.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:23:25.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:23:25.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:23:25.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:23:26.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:23:26.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:23:26.801 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:23:26.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:23:26.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:23:26.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:23:27.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:23:28.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:23:28.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:23:28.820 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:23:28.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:23:28.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:23:28.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:23:29.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:23:30.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:23:30.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:23:30.622 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:23:30.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:23:30.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:23:30.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:23:31.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:23:32.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:23:32.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:23:32.404 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:23:32.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:23:32.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:23:32.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:23:33.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:23:34.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:23:34.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:23:34.224 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:23:34.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:23:34.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:23:34.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:23:34.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:23:35.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:23:35.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:23:35.922 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:23:35.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:23:35.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:23:35.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:23:37.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:23:38.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:23:38.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:23:38.013 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:23:38.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:23:38.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:23:38.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:23:38.475 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2510117920,v1:127.0.0.1:6803/2510117920] [v2:127.0.0.1:6804/2510117920,v1:127.0.0.1:6805/2510117920] exists,up e83d7eeb-41c0-4dc3-a9f9-9349b45d3e5c 2024-06-03T05:23:38.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:23:38.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:23:38.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:23:38.476 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:754: test_run_osd: get_asok_path osd.0 2024-06-03T05:23:38.476 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-03T05:23:38.476 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-03T05:23:38.477 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:23:38.477 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:23:38.477 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:23:38.478 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-osd.0.asok 2024-06-03T05:23:38.478 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:754: test_run_osd: CEPH_ARGS= 2024-06-03T05:23:38.478 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:754: test_run_osd: ceph --format=json daemon /tmp/ceph-asok.70897/ceph-osd.0.asok config get osd_max_backfills 2024-06-03T05:23:38.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:754: test_run_osd: local 'backfills={"osd_max_backfills":"1"}' 2024-06-03T05:23:38.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:755: test_run_osd: echo '{"osd_max_backfills":"1"}' 2024-06-03T05:23:38.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:755: test_run_osd: grep --quiet osd_max_backfills 2024-06-03T05:23:38.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:757: test_run_osd: run_osd td/ceph-helpers 1 --osd-max-backfills 20 2024-06-03T05:23:38.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:23:38.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:23:38.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-03T05:23:38.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:23:38.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-06-03T05:23:38.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:23:38.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:23:38.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:23:38.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:23:38.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-06-03T05:23:38.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-06-03T05:23:38.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:23:38.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:23:38.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:23:38.588 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:23:38.588 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:23:38.588 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:23:38.590 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:23:38.590 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:23:38.590 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:23:38.590 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:23:38.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:23:38.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:23:38.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:23:38.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:23:38.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:23:38.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:23:38.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:23:38.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:23:38.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:23:38.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:23:38.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:23:38.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-max-backfills 20' 2024-06-03T05:23:38.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-06-03T05:23:38.592 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:23:38.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=74b6254f-f845-403b-84d7-0cf6cb379e9b 2024-06-03T05:23:38.592 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 74b6254f-f845-403b-84d7-0cf6cb379e9b 2024-06-03T05:23:38.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 74b6254f-f845-403b-84d7-0cf6cb379e9b' 2024-06-03T05:23:38.592 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:23:38.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBaU11muyFzJBAAAg3G7KlNEUVzteGMSyWcvg== 2024-06-03T05:23:38.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBaU11muyFzJBAAAg3G7KlNEUVzteGMSyWcvg=="}' 2024-06-03T05:23:38.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 74b6254f-f845-403b-84d7-0cf6cb379e9b -i td/ceph-helpers/1/new.json 2024-06-03T05:23:39.187 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:23:39.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-06-03T05:23:39.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 --mkfs --key AQBaU11muyFzJBAAAg3G7KlNEUVzteGMSyWcvg== --osd-uuid 74b6254f-f845-403b-84d7-0cf6cb379e9b 2024-06-03T05:23:39.238 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:23:39.236+0000 7f56fe9867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:23:39.257 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:23:39.256+0000 7f56fe9867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:23:39.259 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:23:39.256+0000 7f56fe9867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:23:39.259 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:23:39.256+0000 7f56fe9867c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-03T05:23:39.259 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:23:39.256+0000 7f56fe9867c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-03T05:23:39.259 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:23:39.256+0000 7f56fe9867c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-03T05:23:39.297 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:23:39.296+0000 7f56fe9867c0 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-06-03T05:23:45.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-06-03T05:23:45.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:23:45.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-03T05:23:45.968 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-06-03T05:23:45.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:23:46.526 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-06-03T05:23:46.526 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-03T05:23:46.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2024-06-03T05:23:46.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:23:46.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:23:46.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:23:46.553 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:23:46.548+0000 7f84e3df37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:23:46.579 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:23:46.576+0000 7f84e3df37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:23:46.581 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:23:46.576+0000 7f84e3df37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:23:46.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-03T05:23:46.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:23:46.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T05:23:46.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:23:46.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:23:46.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:23:46.935 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:23:46.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:23:46.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:23:46.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:23:47.160 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:23:47.156+0000 7f84e3df37c0 -1 Falling back to public interface 2024-06-03T05:23:47.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:23:48.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:23:48.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:23:48.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:23:48.349 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:23:48.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:23:48.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:23:48.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:23:49.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:23:49.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:23:49.759 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:23:49.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:23:49.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:23:49.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:23:49.793 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:23:49.792+0000 7f84e3df37c0 -1 osd.1 0 log_to_monitors true 2024-06-03T05:23:50.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:23:51.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:23:51.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:23:51.318 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:23:51.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:23:51.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:23:51.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:23:51.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:23:52.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:23:52.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:23:52.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:23:52.867 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:23:52.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:23:52.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:23:53.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:23:54.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:23:54.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:23:54.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:23:54.351 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:23:54.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:23:54.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:23:54.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:23:55.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:23:55.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:23:55.801 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:23:55.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:23:55.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:23:55.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:23:56.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:23:57.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:23:57.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:23:57.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:23:57.275 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:23:57.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:23:57.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:23:57.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:23:58.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:23:58.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:23:58.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:23:58.872 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:23:58.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:23:58.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:23:59.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:24:00.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:24:00.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:24:00.333 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:24:00.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:24:00.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:24:00.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:24:00.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:24:01.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:24:01.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:24:01.782 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:24:01.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:24:01.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:24:01.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:24:02.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:24:03.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:24:03.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:24:03.272 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:24:03.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:24:03.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:24:03.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:24:04.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:24:05.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:24:05.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:24:05.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:24:05.610 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:24:05.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:24:05.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:24:06.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:24:07.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:24:07.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:24:07.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:24:07.509 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:24:07.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:24:07.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:24:08.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:24:09.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:24:09.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:24:09.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:24:09.475 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:24:09.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:24:09.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:24:10.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:24:11.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:24:11.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:24:11.886 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:24:11.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:24:11.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:24:11.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:24:12.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:24:13.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:24:13.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:24:13.557 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:24:13.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:24:13.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:24:13.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:24:14.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:24:15.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:24:15.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:24:15.747 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:24:15.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:24:15.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:24:15.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:24:16.269 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/899495574,v1:127.0.0.1:6811/899495574] [v2:127.0.0.1:6812/899495574,v1:127.0.0.1:6813/899495574] exists,up 74b6254f-f845-403b-84d7-0cf6cb379e9b 2024-06-03T05:24:16.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:24:16.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:24:16.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:24:16.270 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:759: test_run_osd: get_asok_path osd.1 2024-06-03T05:24:16.271 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-03T05:24:16.271 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-03T05:24:16.271 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:24:16.271 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:24:16.271 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:24:16.271 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-osd.1.asok 2024-06-03T05:24:16.272 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:759: test_run_osd: CEPH_ARGS= 2024-06-03T05:24:16.272 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:759: test_run_osd: ceph --format=json daemon /tmp/ceph-asok.70897/ceph-osd.1.asok config get osd_max_backfills 2024-06-03T05:24:16.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:759: test_run_osd: local 'backfills={"osd_max_backfills":"20"}' 2024-06-03T05:24:16.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:760: test_run_osd: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2024-06-03T05:24:16.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:762: test_run_osd: CEPH_ARGS='--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30' 2024-06-03T05:24:16.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:762: test_run_osd: run_osd td/ceph-helpers 2 2024-06-03T05:24:16.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:24:16.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:24:16.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-03T05:24:16.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:24:16.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/2 2024-06-03T05:24:16.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30' 2024-06-03T05:24:16.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:24:16.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:24:16.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:24:16.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2024-06-03T05:24:16.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2024-06-03T05:24:16.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:24:16.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:24:16.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:24:16.386 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:24:16.387 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:24:16.387 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:24:16.388 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:24:16.388 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:24:16.388 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:24:16.388 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:24:16.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:24:16.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:24:16.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:24:16.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:24:16.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:24:16.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:24:16.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:24:16.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:24:16.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:24:16.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:24:16.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:24:16.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:24:16.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/2 2024-06-03T05:24:16.390 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:24:16.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=39273674-4576-421c-992b-d4e5e25eefd3 2024-06-03T05:24:16.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 39273674-4576-421c-992b-d4e5e25eefd3' 2024-06-03T05:24:16.390 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 39273674-4576-421c-992b-d4e5e25eefd3 2024-06-03T05:24:16.391 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:24:16.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCAU11mwBiWGBAAnStoVlzIGZYVSQXiEZPeBg== 2024-06-03T05:24:16.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCAU11mwBiWGBAAnStoVlzIGZYVSQXiEZPeBg=="}' 2024-06-03T05:24:16.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 39273674-4576-421c-992b-d4e5e25eefd3 -i td/ceph-helpers/2/new.json 2024-06-03T05:24:16.899 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:24:16.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/2/new.json 2024-06-03T05:24:16.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCAU11mwBiWGBAAnStoVlzIGZYVSQXiEZPeBg== --osd-uuid 39273674-4576-421c-992b-d4e5e25eefd3 2024-06-03T05:24:16.951 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:24:16.948+0000 7f6a603687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:24:16.978 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:24:16.976+0000 7f6a603687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:24:16.979 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:24:16.976+0000 7f6a603687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:24:16.979 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:24:16.976+0000 7f6a603687c0 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-06-03T05:24:16.980 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:24:16.976+0000 7f6a603687c0 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-06-03T05:24:16.980 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:24:16.976+0000 7f6a603687c0 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-06-03T05:24:17.017 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:24:17.016+0000 7f6a603687c0 -1 bluestore(td/ceph-helpers/2) _read_fsid unparsable uuid 2024-06-03T05:24:23.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/2/keyring 2024-06-03T05:24:23.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:24:23.906 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-06-03T05:24:23.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-03T05:24:23.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:24:24.453 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-06-03T05:24:24.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-03T05:24:24.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:24:24.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:24:24.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:24:24.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:24:24.481 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:24:24.476+0000 7f50f61547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:24:24.483 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:24:24.480+0000 7f50f61547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:24:24.484 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:24:24.480+0000 7f50f61547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:24:25.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-03T05:24:25.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:24:25.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T05:24:25.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:24:25.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:24:25.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:24:25.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:24:25.019 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:24:25.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:24:25.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:24:25.026 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:24:25.024+0000 7f50f61547c0 -1 Falling back to public interface 2024-06-03T05:24:25.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:24:26.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:24:26.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:24:26.490 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:24:26.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:24:26.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:24:26.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:24:26.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:24:27.711 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:24:27.708+0000 7f50f61547c0 -1 osd.2 0 log_to_monitors true 2024-06-03T05:24:27.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:24:27.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:24:27.902 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:24:27.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:24:27.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:24:27.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:24:28.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:24:29.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:24:29.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:24:29.635 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:24:29.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:24:29.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:24:29.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:24:30.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:24:31.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:24:31.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:24:31.333 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:24:31.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:24:31.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:24:31.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:24:32.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:24:33.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:24:33.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:24:33.009 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:24:33.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:24:33.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:24:33.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:24:33.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:24:34.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:24:34.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:24:34.483 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:24:34.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:24:34.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:24:34.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:24:35.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:24:36.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:24:36.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:24:36.086 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:24:36.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:24:36.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:24:36.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:24:36.526 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:24:37.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:24:37.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:24:37.528 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:24:37.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:24:37.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:24:37.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:24:38.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:24:39.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:24:39.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:24:39.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:24:39.336 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:24:39.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:24:39.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:24:40.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:24:41.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:24:41.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:24:41.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:24:41.077 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:24:41.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:24:41.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:24:41.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:24:42.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:24:42.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:24:42.516 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:24:42.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:24:42.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:24:42.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:24:43.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:24:44.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:24:44.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:24:44.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:24:44.265 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:24:44.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:24:44.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:24:44.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:24:45.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:24:45.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:24:45.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:24:45.829 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:24:45.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:24:45.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:24:47.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:24:48.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:24:48.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:24:48.124 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:24:48.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:24:48.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:24:48.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:24:49.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:24:50.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:24:50.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:24:50.039 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:24:50.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:24:50.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:24:50.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:24:50.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:24:51.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:24:51.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:24:51.617 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:24:51.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:24:51.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:24:51.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:24:52.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:24:53.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:24:53.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:24:53.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:24:53.729 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:24:53.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:24:53.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:24:54.261 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/968301583,v1:127.0.0.1:6819/968301583] [v2:127.0.0.1:6820/968301583,v1:127.0.0.1:6821/968301583] exists,up 39273674-4576-421c-992b-d4e5e25eefd3 2024-06-03T05:24:54.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:24:54.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:24:54.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:24:54.263 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:764: test_run_osd: get_asok_path osd.2 2024-06-03T05:24:54.263 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-06-03T05:24:54.263 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-06-03T05:24:54.264 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:24:54.264 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:24:54.264 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:24:54.264 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-osd.2.asok 2024-06-03T05:24:54.265 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:764: test_run_osd: CEPH_ARGS= 2024-06-03T05:24:54.265 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:764: test_run_osd: ceph --format=json daemon /tmp/ceph-asok.70897/ceph-osd.2.asok config get osd_max_backfills 2024-06-03T05:24:54.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:764: test_run_osd: local 'backfills={"osd_max_backfills":"30"}' 2024-06-03T05:24:54.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:765: test_run_osd: test '{"osd_max_backfills":"30"}' = '{"osd_max_backfills":"30"}' 2024-06-03T05:24:54.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:767: test_run_osd: teardown td/ceph-helpers 2024-06-03T05:24:54.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:24:54.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:24:54.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:24:54.391 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:24:54.391 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:24:54.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:24:54.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:24:54.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:24:54.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:24:54.703 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:24:54.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:24:54.705 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:24:54.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:24:54.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:24:54.707 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:24:54.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:24:54.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:24:54.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:24:54.709 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:24:54.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:24:54.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:24:54.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:24:54.728 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:24:54.728 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:24:54.728 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:24:54.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:24:54.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:24:54.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:24:54.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:24:54.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_set_config td/ceph-helpers 2024-06-03T05:24:54.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: test_set_config: local dir=td/ceph-helpers 2024-06-03T05:24:54.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: test_set_config: setup td/ceph-helpers 2024-06-03T05:24:54.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-03T05:24:54.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-03T05:24:54.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:24:54.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:24:54.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:24:54.731 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:24:54.731 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:24:54.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:24:54.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:24:54.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:24:54.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:24:54.733 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:24:54.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:24:54.734 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:24:54.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:24:54.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:24:54.736 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:24:54.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:24:54.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:24:54.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:24:54.738 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:24:54.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:24:54.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:24:54.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:24:54.741 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:24:54.741 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:24:54.741 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:24:54.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:24:54.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:24:54.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:24:54.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-03T05:24:54.744 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:24:54.744 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:24:54.744 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:24:54.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70897 2024-06-03T05:24:54.746 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:24:54.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:24:54.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:24:54.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-03T05:24:54.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: test_set_config: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:24:54.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-03T05:24:54.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:24:54.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:24:54.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:24:54.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-03T05:24:54.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:24:55.160 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:24:55.160 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:24:55.160 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:24:55.161 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:24:55.161 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:24:55.161 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:24:55.161 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:24:55.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:24:55.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:24:55.454 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:24:55.454 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:24:55.455 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:24:55.455 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:24:55.455 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:24:55.455 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:24:55.456 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:24:55.456 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:24:55.456 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:24:55.456 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:24:55.456 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:24:55.456 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:24:55.457 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:24:55.457 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get fsid 2024-06-03T05:24:55.573 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:24:55.573 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:24:55.573 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:24:55.573 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:24:55.574 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:24:55.574 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:24:55.574 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:24:55.575 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:24:55.575 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:24:55.575 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:24:55.575 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:24:55.575 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:24:55.575 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:24:55.575 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get mon_host 2024-06-03T05:24:55.682 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: test_set_config: get_config mon a ms_crc_header 2024-06-03T05:24:55.683 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:24:55.683 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:24:55.683 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=ms_crc_header 2024-06-03T05:24:55.683 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .ms_crc_header 2024-06-03T05:24:55.683 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:24:55.684 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:24:55.684 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:24:55.684 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:24:55.684 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:24:55.685 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:24:55.685 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:24:55.685 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:24:55.685 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get ms_crc_header 2024-06-03T05:24:55.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: test_set_config: test true = true 2024-06-03T05:24:55.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1177: test_set_config: set_config mon a ms_crc_header false 2024-06-03T05:24:55.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=mon 2024-06-03T05:24:55.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=a 2024-06-03T05:24:55.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=ms_crc_header 2024-06-03T05:24:55.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=false 2024-06-03T05:24:55.791 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-06-03T05:24:55.791 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path mon.a 2024-06-03T05:24:55.791 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:24:55.792 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:24:55.792 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:24:55.792 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:24:55.792 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:24:55.792 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:24:55.793 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config set ms_crc_header false 2024-06-03T05:24:55.903 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-06-03T05:24:55.904 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1178: test_set_config: get_config mon a ms_crc_header 2024-06-03T05:24:55.904 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:24:55.904 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:24:55.904 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=ms_crc_header 2024-06-03T05:24:55.905 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .ms_crc_header 2024-06-03T05:24:55.905 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:24:55.905 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:24:55.905 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:24:55.905 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:24:55.905 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:24:55.905 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:24:55.906 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:24:55.906 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:24:55.906 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get ms_crc_header 2024-06-03T05:24:56.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1178: test_set_config: test false = false 2024-06-03T05:24:56.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1179: test_set_config: set_config mon a ms_crc_header true 2024-06-03T05:24:56.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=mon 2024-06-03T05:24:56.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=a 2024-06-03T05:24:56.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=ms_crc_header 2024-06-03T05:24:56.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-06-03T05:24:56.009 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-06-03T05:24:56.009 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path mon.a 2024-06-03T05:24:56.009 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:24:56.009 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:24:56.009 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:24:56.009 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:24:56.009 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:24:56.009 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:24:56.009 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config set ms_crc_header true 2024-06-03T05:24:56.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-06-03T05:24:56.108 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1180: test_set_config: get_config mon a ms_crc_header 2024-06-03T05:24:56.108 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:24:56.108 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:24:56.108 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=ms_crc_header 2024-06-03T05:24:56.110 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:24:56.110 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:24:56.110 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .ms_crc_header 2024-06-03T05:24:56.110 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:24:56.110 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:24:56.110 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:24:56.110 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:24:56.110 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:24:56.110 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:24:56.110 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get ms_crc_header 2024-06-03T05:24:56.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1180: test_set_config: test true = true 2024-06-03T05:24:56.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1181: test_set_config: teardown td/ceph-helpers 2024-06-03T05:24:56.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:24:56.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:24:56.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:24:56.210 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:24:56.210 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:24:56.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:24:56.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:24:56.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:24:56.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:24:56.316 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:24:56.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:24:56.318 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:24:56.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:24:56.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:24:56.320 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:24:56.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:24:56.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:24:56.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:24:56.322 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:24:56.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:24:56.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:24:56.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:24:56.326 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:24:56.326 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:24:56.326 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:24:56.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:24:56.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:24:56.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:24:56.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:24:56.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_setup td/ceph-helpers 2024-06-03T05:24:56.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:144: test_setup: local dir=td/ceph-helpers 2024-06-03T05:24:56.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:145: test_setup: setup td/ceph-helpers 2024-06-03T05:24:56.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-03T05:24:56.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-03T05:24:56.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:24:56.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:24:56.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:24:56.329 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:24:56.329 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:24:56.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:24:56.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:24:56.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:24:56.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:24:56.330 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:24:56.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:24:56.332 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:24:56.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:24:56.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:24:56.333 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:24:56.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:24:56.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:24:56.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:24:56.335 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:24:56.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:24:56.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:24:56.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:24:56.338 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:24:56.338 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:24:56.338 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:24:56.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:24:56.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:24:56.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:24:56.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-03T05:24:56.341 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:24:56.342 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:24:56.342 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:24:56.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70897 2024-06-03T05:24:56.344 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:24:56.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:24:56.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:24:56.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-03T05:24:56.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:146: test_setup: test -d td/ceph-helpers 2024-06-03T05:24:56.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:147: test_setup: setup td/ceph-helpers 2024-06-03T05:24:56.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-03T05:24:56.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-03T05:24:56.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:24:56.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:24:56.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:24:56.345 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:24:56.345 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:24:56.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:24:56.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:24:56.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:24:56.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:24:56.346 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:24:56.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:24:56.347 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:24:56.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:24:56.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:24:56.349 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:24:56.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:24:56.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:24:56.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:24:56.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:24:56.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:24:56.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:24:56.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:24:56.354 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:24:56.354 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:24:56.354 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:24:56.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:24:56.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:24:56.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:24:56.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-03T05:24:56.356 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:24:56.356 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:24:56.356 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:24:56.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70897 2024-06-03T05:24:56.359 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:24:56.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:24:56.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:24:56.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-03T05:24:56.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:148: test_setup: test -d td/ceph-helpers 2024-06-03T05:24:56.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:149: test_setup: teardown td/ceph-helpers 2024-06-03T05:24:56.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:24:56.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:24:56.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:24:56.360 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:24:56.360 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:24:56.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:24:56.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:24:56.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:24:56.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:24:56.361 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:24:56.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:24:56.363 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:24:56.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:24:56.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:24:56.364 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:24:56.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:24:56.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:24:56.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:24:56.366 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:24:56.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:24:56.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:24:56.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:24:56.369 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:24:56.369 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:24:56.369 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:24:56.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:24:56.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:24:56.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:24:56.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:24:56.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_teardown td/ceph-helpers 2024-06-03T05:24:56.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:220: test_teardown: local dir=td/ceph-helpers 2024-06-03T05:24:56.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:221: test_teardown: setup td/ceph-helpers 2024-06-03T05:24:56.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-03T05:24:56.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-03T05:24:56.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:24:56.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:24:56.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:24:56.372 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:24:56.372 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:24:56.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:24:56.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:24:56.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:24:56.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:24:56.373 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:24:56.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:24:56.374 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:24:56.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:24:56.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:24:56.376 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:24:56.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:24:56.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:24:56.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:24:56.377 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:24:56.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:24:56.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:24:56.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:24:56.381 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:24:56.381 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:24:56.381 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:24:56.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:24:56.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:24:56.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:24:56.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-03T05:24:56.384 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:24:56.384 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:24:56.384 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:24:56.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70897 2024-06-03T05:24:56.386 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:24:56.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:24:56.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:24:56.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-03T05:24:56.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:222: test_teardown: teardown td/ceph-helpers 2024-06-03T05:24:56.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:24:56.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:24:56.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:24:56.387 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:24:56.387 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:24:56.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:24:56.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:24:56.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:24:56.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:24:56.389 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:24:56.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:24:56.390 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:24:56.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:24:56.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:24:56.391 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:24:56.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:24:56.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:24:56.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:24:56.393 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:24:56.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:24:56.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:24:56.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:24:56.396 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:24:56.396 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:24:56.396 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:24:56.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:24:56.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:24:56.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:24:56.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:223: test_teardown: test -d td/ceph-helpers 2024-06-03T05:24:56.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:24:56.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_wait_background td/ceph-helpers 2024-06-03T05:24:56.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: test_wait_background: local pids= 2024-06-03T05:24:56.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: test_wait_background: run_in_background pids bash -c 'sleep 1; exit 1' 2024-06-03T05:24:56.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T05:24:56.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T05:24:56.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 143569"' 2024-06-03T05:24:56.400 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 143569' 2024-06-03T05:24:56.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: test_wait_background: run_in_background pids bash -c 'sleep 2; exit 0' 2024-06-03T05:24:56.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T05:24:56.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T05:24:56.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 143570"' 2024-06-03T05:24:56.401 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 143570' 2024-06-03T05:24:56.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: test_wait_background: wait_background pids 2024-06-03T05:24:56.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 143569 143570' 2024-06-03T05:24:56.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-06-03T05:24:56.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: bash -c 'sleep 1; exit 1' 2024-06-03T05:24:56.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-03T05:24:56.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 143569 2024-06-03T05:24:56.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/143572: /' 2024-06-03T05:24:56.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: bash -c 'sleep 2; exit 0' 2024-06-03T05:24:56.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/143574: /' 2024-06-03T05:24:57.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 1 2024-06-03T05:24:57.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: wait_background: return_code=1 2024-06-03T05:24:57.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-03T05:24:57.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 143570 2024-06-03T05:24:58.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T05:24:58.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-06-03T05:24:58.404 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-06-03T05:24:58.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 1 2024-06-03T05:24:58.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: test_wait_background: '[' 1 -ne 1 ']' 2024-06-03T05:24:58.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: test_wait_background: run_in_background pids bash -c 'sleep 1; exit 0' 2024-06-03T05:24:58.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T05:24:58.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T05:24:58.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 143577"' 2024-06-03T05:24:58.404 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 143577' 2024-06-03T05:24:58.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: test_wait_background: run_in_background pids bash -c 'sleep 2; exit 0' 2024-06-03T05:24:58.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-03T05:24:58.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-03T05:24:58.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: bash -c 'sleep 1; exit 0' 2024-06-03T05:24:58.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 143579"' 2024-06-03T05:24:58.405 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 143579' 2024-06-03T05:24:58.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2189: test_wait_background: wait_background pids 2024-06-03T05:24:58.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 143577 143579' 2024-06-03T05:24:58.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-06-03T05:24:58.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-03T05:24:58.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 143577 2024-06-03T05:24:58.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/143580: /' 2024-06-03T05:24:58.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: bash -c 'sleep 2; exit 0' 2024-06-03T05:24:58.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/143582: /' 2024-06-03T05:24:59.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T05:24:59.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-03T05:24:59.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 143579 2024-06-03T05:25:00.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-03T05:25:00.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-06-03T05:25:00.408 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-06-03T05:25:00.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-06-03T05:25:00.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: test_wait_background: '[' 0 -ne 0 ']' 2024-06-03T05:25:00.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2192: test_wait_background: '[' '!' -z '' ']' 2024-06-03T05:25:00.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:25:00.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_wait_for_clean td/ceph-helpers 2024-06-03T05:25:00.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1677: test_wait_for_clean: local dir=td/ceph-helpers 2024-06-03T05:25:00.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1679: test_wait_for_clean: setup td/ceph-helpers 2024-06-03T05:25:00.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-03T05:25:00.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-03T05:25:00.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:25:00.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:25:00.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:25:00.409 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:25:00.409 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:25:00.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:25:00.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:25:00.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:25:00.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:25:00.413 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:25:00.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:25:00.414 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:25:00.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:25:00.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:25:00.416 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:25:00.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:25:00.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:25:00.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:25:00.418 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:25:00.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:25:00.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:25:00.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:25:00.422 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:25:00.422 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:25:00.422 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:25:00.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:25:00.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:25:00.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:25:00.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-03T05:25:00.425 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:25:00.425 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:25:00.425 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:25:00.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70897 2024-06-03T05:25:00.427 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:25:00.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:25:00.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:25:00.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-03T05:25:00.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1680: test_wait_for_clean: run_mon td/ceph-helpers a --osd_pool_default_size=2 2024-06-03T05:25:00.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-03T05:25:00.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:25:00.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:25:00.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:25:00.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-03T05:25:00.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2024-06-03T05:25:00.851 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:25:00.851 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:25:00.852 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:25:00.852 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:25:00.852 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:25:00.852 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:25:00.852 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:25:00.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-06-03T05:25:01.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:25:01.072 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:25:01.072 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:25:01.072 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:25:01.072 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:25:01.072 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:25:01.073 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:25:01.073 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:25:01.073 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:25:01.074 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:25:01.074 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:25:01.074 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:25:01.074 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:25:01.074 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:25:01.074 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get fsid 2024-06-03T05:25:01.189 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:25:01.189 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:25:01.190 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:25:01.190 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:25:01.190 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:25:01.191 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:25:01.191 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:25:01.191 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:25:01.191 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:25:01.191 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:25:01.191 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:25:01.191 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:25:01.191 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:25:01.191 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get mon_host 2024-06-03T05:25:01.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1681: test_wait_for_clean: run_osd td/ceph-helpers 0 2024-06-03T05:25:01.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:25:01.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:25:01.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T05:25:01.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:25:01.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-03T05:25:01.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:25:01.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:25:01.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:25:01.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:25:01.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-03T05:25:01.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-03T05:25:01.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:25:01.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:25:01.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:25:01.309 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:25:01.309 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:25:01.309 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:25:01.309 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:25:01.309 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:25:01.309 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:25:01.310 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:25:01.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:25:01.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:25:01.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:25:01.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:25:01.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:25:01.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:25:01.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:25:01.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:25:01.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:25:01.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:25:01.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:25:01.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:25:01.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-03T05:25:01.313 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:25:01.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ff814cf5-edfb-4904-bc3c-e95d281ff14b 2024-06-03T05:25:01.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ff814cf5-edfb-4904-bc3c-e95d281ff14b' 2024-06-03T05:25:01.314 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 ff814cf5-edfb-4904-bc3c-e95d281ff14b 2024-06-03T05:25:01.314 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:25:01.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCtU11m6+7DExAA3dky4sRbVq22udfAMrQ5HA== 2024-06-03T05:25:01.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCtU11m6+7DExAA3dky4sRbVq22udfAMrQ5HA=="}' 2024-06-03T05:25:01.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ff814cf5-edfb-4904-bc3c-e95d281ff14b -i td/ceph-helpers/0/new.json 2024-06-03T05:25:02.283 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:25:02.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-03T05:25:02.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCtU11m6+7DExAA3dky4sRbVq22udfAMrQ5HA== --osd-uuid ff814cf5-edfb-4904-bc3c-e95d281ff14b 2024-06-03T05:25:02.338 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:25:02.336+0000 7f0d221737c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:25:02.369 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:25:02.364+0000 7f0d221737c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:25:02.370 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:25:02.368+0000 7f0d221737c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:25:02.371 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:25:02.368+0000 7f0d221737c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:25:02.371 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:25:02.368+0000 7f0d221737c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:25:02.371 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:25:02.368+0000 7f0d221737c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:25:02.408 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:25:02.404+0000 7f0d221737c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-03T05:25:08.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-03T05:25:08.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:25:08.570 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T05:25:08.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T05:25:08.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:25:08.820 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:25:08.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T05:25:08.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:25:08.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:25:08.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:25:08.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:25:08.852 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:25:08.848+0000 7f1d49efe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:25:08.855 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:25:08.852+0000 7f1d49efe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:25:08.856 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:25:08.852+0000 7f1d49efe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:25:09.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T05:25:09.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:25:09.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:25:09.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:25:09.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:25:09.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:25:09.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:25:09.051 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:25:09.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:25:09.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:25:09.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:25:09.404 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:25:09.400+0000 7f1d49efe7c0 -1 Falling back to public interface 2024-06-03T05:25:10.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:25:10.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:25:10.264 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:25:10.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:25:10.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:25:10.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:25:10.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:25:11.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:25:11.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:25:11.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:25:11.506 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:25:11.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:25:11.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:25:11.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:25:12.073 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:25:12.072+0000 7f1d49efe7c0 -1 osd.0 0 log_to_monitors true 2024-06-03T05:25:12.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:25:12.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:25:12.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:25:12.743 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:25:12.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:25:12.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:25:12.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:25:13.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:25:13.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:25:13.969 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:25:13.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:25:13.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:25:13.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:25:14.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:25:15.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:25:15.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:25:15.209 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:25:15.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:25:15.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:25:15.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:25:15.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:25:16.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:25:16.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:25:16.449 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:25:16.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:25:16.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:25:16.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:25:16.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:25:17.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:25:17.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:25:17.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:25:17.690 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:25:17.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:25:17.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:25:17.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:25:18.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:25:18.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:25:18.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:25:18.921 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:25:18.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:25:18.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:25:19.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:25:20.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:25:20.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:25:20.156 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:25:20.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:25:20.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:25:20.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:25:20.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:25:21.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:25:21.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:25:21.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:25:21.391 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:25:21.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:25:21.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:25:21.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:25:22.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:25:22.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:25:22.631 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:25:22.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:25:22.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:25:22.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:25:22.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:25:23.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:25:23.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:25:23.866 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:25:23.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:25:23.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:25:23.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:25:24.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:25:25.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:25:25.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:25:25.094 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:25:25.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:25:25.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:25:25.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:25:25.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:25:26.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:25:26.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:25:26.334 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:25:26.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:25:26.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:25:26.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:25:26.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:25:27.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:25:27.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:25:27.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:25:27.572 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:25:27.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:25:27.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:25:27.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:25:28.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:25:28.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:25:28.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:25:28.812 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:25:28.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:25:28.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:25:29.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:25:30.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:25:30.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:25:30.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:25:30.043 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:25:30.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:25:30.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:25:30.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:25:31.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:25:31.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:25:31.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T05:25:31.276 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-06-03T05:25:31.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:25:31.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:25:31.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:25:32.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:25:32.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:25:32.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-03T05:25:32.517 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-06-03T05:25:32.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:25:32.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:25:32.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:25:33.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:25:33.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:25:33.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-03T05:25:33.754 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-06-03T05:25:33.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:25:33.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:25:33.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:25:34.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:25:34.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:25:34.989 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-06-03T05:25:34.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-03T05:25:34.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:25:34.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:25:35.259 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2780887385,v1:127.0.0.1:6801/2780887385] [v2:127.0.0.1:6802/2780887385,v1:127.0.0.1:6803/2780887385] exists,up ff814cf5-edfb-4904-bc3c-e95d281ff14b 2024-06-03T05:25:35.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:25:35.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:25:35.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:25:35.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1682: test_wait_for_clean: run_mgr td/ceph-helpers x 2024-06-03T05:25:35.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-03T05:25:35.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T05:25:35.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T05:25:35.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T05:25:35.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-03T05:25:35.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T05:25:35.726 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T05:25:35.727 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:25:35.727 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:25:35.727 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:25:35.727 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:25:35.727 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:25:35.727 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:25:35.728 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:25:35.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:25:35.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1683: test_wait_for_clean: create_rbd_pool 2024-06-03T05:25:35.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-03T05:25:36.038 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' does not exist 2024-06-03T05:25:36.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-03T05:25:36.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-03T05:25:36.429 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' already exists 2024-06-03T05:25:36.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T05:25:37.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-03T05:25:38.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1684: test_wait_for_clean: WAIT_FOR_CLEAN_TIMEOUT=1 2024-06-03T05:25:38.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1684: test_wait_for_clean: wait_for_clean 2024-06-03T05:25:38.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T05:25:38.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T05:25:38.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T05:25:38.460 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 1 .1 2024-06-03T05:25:38.461 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:25:38.462 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:25:38.462 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:25:38.462 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:25:38.462 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:25:38.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.3') 2024-06-03T05:25:38.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T05:25:38.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T05:25:38.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T05:25:38.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T05:25:38.492 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T05:25:38.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-06-03T05:25:38.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T05:25:38.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:25:38.766 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T05:25:38.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=0 2024-06-03T05:25:38.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 0 2024-06-03T05:25:38.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-0' 2024-06-03T05:25:38.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:25:38.920 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-0 2024-06-03T05:25:38.920 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:25:38.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T05:25:38.922 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-0 2024-06-03T05:25:38.922 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:25:38.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=0 2024-06-03T05:25:38.923 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 0 2024-06-03T05:25:38.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 0' 2024-06-03T05:25:38.923 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:25:39.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 0 -lt 0 2024-06-03T05:25:39.162 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T05:25:39.163 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:25:39.163 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:25:39.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 == 0 2024-06-03T05:25:39.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: sleep 1 2024-06-03T05:25:40.404 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T05:25:40.404 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:25:40.405 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:25:40.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 == 0 2024-06-03T05:25:40.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: sleep 1 2024-06-03T05:25:41.719 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T05:25:41.720 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:25:41.720 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:25:42.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-03T05:25:42.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T05:25:42.263 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T05:25:42.264 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T05:25:42.264 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T05:25:42.264 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T05:25:42.264 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T05:25:42.264 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T05:25:42.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-03T05:25:42.697 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T05:25:42.698 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:25:42.698 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:25:43.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-06-03T05:25:43.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-06-03T05:25:43.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-03T05:25:43.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-06-03T05:25:43.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T05:25:43.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-03T05:25:43.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T05:25:43.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T05:25:43.315 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T05:25:43.315 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T05:25:43.315 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T05:25:43.315 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T05:25:43.316 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T05:25:43.316 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T05:25:43.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-03T05:25:43.704 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T05:25:43.704 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:25:43.704 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:25:44.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-06-03T05:25:44.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-06-03T05:25:44.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-03T05:25:44.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T05:25:44.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T05:25:44.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T05:25:44.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T05:25:44.231 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T05:25:44.231 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T05:25:44.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-03T05:25:44.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-03T05:25:44.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 4 )) 2024-06-03T05:25:44.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T05:25:44.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-03T05:25:44.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T05:25:44.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T05:25:44.995 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T05:25:44.995 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T05:25:44.995 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T05:25:44.995 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T05:25:44.996 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T05:25:44.996 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T05:25:45.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-03T05:25:45.396 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T05:25:45.396 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:25:45.396 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:25:45.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-06-03T05:25:45.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-06-03T05:25:45.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-03T05:25:45.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T05:25:45.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T05:25:45.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T05:25:45.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T05:25:45.923 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T05:25:45.923 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T05:25:46.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-03T05:25:46.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-03T05:25:46.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 4 )) 2024-06-03T05:25:46.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T05:25:46.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-06-03T05:25:47.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T05:25:47.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T05:25:47.010 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T05:25:47.010 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T05:25:47.011 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T05:25:47.011 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T05:25:47.011 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T05:25:47.011 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T05:25:47.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-03T05:25:47.412 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T05:25:47.412 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:25:47.412 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:25:47.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-06-03T05:25:47.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-06-03T05:25:47.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-03T05:25:47.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T05:25:47.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T05:25:47.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T05:25:47.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T05:25:47.934 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T05:25:47.935 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T05:25:48.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-03T05:25:48.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-03T05:25:48.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 4 )) 2024-06-03T05:25:48.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T05:25:48.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.3 2024-06-03T05:25:48.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T05:25:48.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T05:25:48.851 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T05:25:48.851 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T05:25:48.851 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T05:25:48.851 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T05:25:48.852 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T05:25:48.852 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T05:25:49.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-03T05:25:49.248 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T05:25:49.248 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:25:49.249 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:25:49.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-06-03T05:25:49.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-06-03T05:25:49.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-03T05:25:49.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T05:25:49.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T05:25:49.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T05:25:49.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T05:25:49.768 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T05:25:49.768 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T05:25:50.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-03T05:25:50.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-03T05:25:50.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 4 )) 2024-06-03T05:25:50.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-06-03T05:25:50.822 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-06-03T05:25:50.823 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_fingerprint": "4aa6eb78-12c3-4d54-acd0-3a9c40e61576", 2024-06-03T05:25:50.823 INFO:tasks.workunit.client.0.smithi088.stdout: "version": "19.0.0-2946-g3d89accf", 2024-06-03T05:25:50.823 INFO:tasks.workunit.client.0.smithi088.stdout: "commit": "3d89accfe3201286f000e065a1faea2a88c7a6ba", 2024-06-03T05:25:50.823 INFO:tasks.workunit.client.0.smithi088.stdout: "timestamp": "2024-06-03T05:25:50.819442+0000", 2024-06-03T05:25:50.823 INFO:tasks.workunit.client.0.smithi088.stdout: "tag": "", 2024-06-03T05:25:50.823 INFO:tasks.workunit.client.0.smithi088.stdout: "health": { 2024-06-03T05:25:50.823 INFO:tasks.workunit.client.0.smithi088.stdout: "status": "HEALTH_WARN", 2024-06-03T05:25:50.823 INFO:tasks.workunit.client.0.smithi088.stdout: "checks": { 2024-06-03T05:25:50.823 INFO:tasks.workunit.client.0.smithi088.stdout: "PG_DEGRADED": { 2024-06-03T05:25:50.823 INFO:tasks.workunit.client.0.smithi088.stdout: "severity": "HEALTH_WARN", 2024-06-03T05:25:50.823 INFO:tasks.workunit.client.0.smithi088.stdout: "summary": { 2024-06-03T05:25:50.823 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "Degraded data redundancy: 1/2 objects degraded (50.000%), 1 pg degraded", 2024-06-03T05:25:50.823 INFO:tasks.workunit.client.0.smithi088.stdout: "count": 1 2024-06-03T05:25:50.824 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.824 INFO:tasks.workunit.client.0.smithi088.stdout: "detail": [ 2024-06-03T05:25:50.824 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.824 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0]" 2024-06-03T05:25:50.824 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:25:50.824 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-06-03T05:25:50.824 INFO:tasks.workunit.client.0.smithi088.stdout: "muted": false 2024-06-03T05:25:50.824 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.824 INFO:tasks.workunit.client.0.smithi088.stdout: "TOO_FEW_OSDS": { 2024-06-03T05:25:50.824 INFO:tasks.workunit.client.0.smithi088.stdout: "severity": "HEALTH_WARN", 2024-06-03T05:25:50.824 INFO:tasks.workunit.client.0.smithi088.stdout: "summary": { 2024-06-03T05:25:50.824 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "OSD count 1 < osd_pool_default_size 3", 2024-06-03T05:25:50.824 INFO:tasks.workunit.client.0.smithi088.stdout: "count": 2 2024-06-03T05:25:50.825 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.825 INFO:tasks.workunit.client.0.smithi088.stdout: "detail": [], 2024-06-03T05:25:50.825 INFO:tasks.workunit.client.0.smithi088.stdout: "muted": false 2024-06-03T05:25:50.825 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:25:50.825 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.825 INFO:tasks.workunit.client.0.smithi088.stdout: "mutes": [] 2024-06-03T05:25:50.825 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.825 INFO:tasks.workunit.client.0.smithi088.stdout: "monmap_first_committed": 1, 2024-06-03T05:25:50.825 INFO:tasks.workunit.client.0.smithi088.stdout: "monmap_last_committed": 1, 2024-06-03T05:25:50.825 INFO:tasks.workunit.client.0.smithi088.stdout: "monmap": { 2024-06-03T05:25:50.825 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 1, 2024-06-03T05:25:50.825 INFO:tasks.workunit.client.0.smithi088.stdout: "fsid": "213fb4bd-379a-40f7-a383-f6c59451eb9c", 2024-06-03T05:25:50.825 INFO:tasks.workunit.client.0.smithi088.stdout: "modified": "2024-06-03T05:25:00.458076Z", 2024-06-03T05:25:50.826 INFO:tasks.workunit.client.0.smithi088.stdout: "created": "2024-06-03T05:25:00.458076Z", 2024-06-03T05:25:50.826 INFO:tasks.workunit.client.0.smithi088.stdout: "min_mon_release": 19, 2024-06-03T05:25:50.826 INFO:tasks.workunit.client.0.smithi088.stdout: "min_mon_release_name": "squid", 2024-06-03T05:25:50.826 INFO:tasks.workunit.client.0.smithi088.stdout: "election_strategy": 1, 2024-06-03T05:25:50.826 INFO:tasks.workunit.client.0.smithi088.stdout: "disallowed_leaders: ": "", 2024-06-03T05:25:50.826 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_mode": false, 2024-06-03T05:25:50.826 INFO:tasks.workunit.client.0.smithi088.stdout: "tiebreaker_mon": "", 2024-06-03T05:25:50.826 INFO:tasks.workunit.client.0.smithi088.stdout: "removed_ranks: ": "", 2024-06-03T05:25:50.826 INFO:tasks.workunit.client.0.smithi088.stdout: "features": { 2024-06-03T05:25:50.826 INFO:tasks.workunit.client.0.smithi088.stdout: "persistent": [ 2024-06-03T05:25:50.826 INFO:tasks.workunit.client.0.smithi088.stdout: "kraken", 2024-06-03T05:25:50.826 INFO:tasks.workunit.client.0.smithi088.stdout: "luminous", 2024-06-03T05:25:50.826 INFO:tasks.workunit.client.0.smithi088.stdout: "mimic", 2024-06-03T05:25:50.826 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap-prune", 2024-06-03T05:25:50.827 INFO:tasks.workunit.client.0.smithi088.stdout: "nautilus", 2024-06-03T05:25:50.827 INFO:tasks.workunit.client.0.smithi088.stdout: "octopus", 2024-06-03T05:25:50.827 INFO:tasks.workunit.client.0.smithi088.stdout: "pacific", 2024-06-03T05:25:50.827 INFO:tasks.workunit.client.0.smithi088.stdout: "elector-pinging", 2024-06-03T05:25:50.827 INFO:tasks.workunit.client.0.smithi088.stdout: "quincy", 2024-06-03T05:25:50.827 INFO:tasks.workunit.client.0.smithi088.stdout: "reef", 2024-06-03T05:25:50.827 INFO:tasks.workunit.client.0.smithi088.stdout: "squid" 2024-06-03T05:25:50.827 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-06-03T05:25:50.827 INFO:tasks.workunit.client.0.smithi088.stdout: "optional": [] 2024-06-03T05:25:50.827 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.827 INFO:tasks.workunit.client.0.smithi088.stdout: "mons": [ 2024-06-03T05:25:50.827 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.827 INFO:tasks.workunit.client.0.smithi088.stdout: "rank": 0, 2024-06-03T05:25:50.828 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "a", 2024-06-03T05:25:50.828 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addrs": { 2024-06-03T05:25:50.828 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-06-03T05:25:50.828 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.828 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-06-03T05:25:50.828 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:7109", 2024-06-03T05:25:50.828 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 0 2024-06-03T05:25:50.828 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:25:50.828 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-06-03T05:25:50.828 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.828 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:7109/0", 2024-06-03T05:25:50.828 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addr": "127.0.0.1:7109/0", 2024-06-03T05:25:50.828 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 0, 2024-06-03T05:25:50.828 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 0, 2024-06-03T05:25:50.829 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_location": "{}" 2024-06-03T05:25:50.829 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:25:50.829 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-06-03T05:25:50.829 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.829 INFO:tasks.workunit.client.0.smithi088.stdout: "quorum": [ 2024-06-03T05:25:50.829 INFO:tasks.workunit.client.0.smithi088.stdout: 0 2024-06-03T05:25:50.829 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-06-03T05:25:50.829 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap": { 2024-06-03T05:25:50.829 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 12, 2024-06-03T05:25:50.829 INFO:tasks.workunit.client.0.smithi088.stdout: "fsid": "213fb4bd-379a-40f7-a383-f6c59451eb9c", 2024-06-03T05:25:50.829 INFO:tasks.workunit.client.0.smithi088.stdout: "created": "2024-06-03T05:25:01.437376+0000", 2024-06-03T05:25:50.829 INFO:tasks.workunit.client.0.smithi088.stdout: "modified": "2024-06-03T05:25:38.176565+0000", 2024-06-03T05:25:50.829 INFO:tasks.workunit.client.0.smithi088.stdout: "last_up_change": "2024-06-03T05:25:34.698641+0000", 2024-06-03T05:25:50.829 INFO:tasks.workunit.client.0.smithi088.stdout: "last_in_change": "2024-06-03T05:25:02.142853+0000", 2024-06-03T05:25:50.830 INFO:tasks.workunit.client.0.smithi088.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-06-03T05:25:50.830 INFO:tasks.workunit.client.0.smithi088.stdout: "flags_num": 5799936, 2024-06-03T05:25:50.830 INFO:tasks.workunit.client.0.smithi088.stdout: "flags_set": [ 2024-06-03T05:25:50.830 INFO:tasks.workunit.client.0.smithi088.stdout: "pglog_hardlimit", 2024-06-03T05:25:50.830 INFO:tasks.workunit.client.0.smithi088.stdout: "purged_snapdirs", 2024-06-03T05:25:50.830 INFO:tasks.workunit.client.0.smithi088.stdout: "recovery_deletes", 2024-06-03T05:25:50.830 INFO:tasks.workunit.client.0.smithi088.stdout: "sortbitwise" 2024-06-03T05:25:50.830 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-06-03T05:25:50.830 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_version": 3, 2024-06-03T05:25:50.830 INFO:tasks.workunit.client.0.smithi088.stdout: "full_ratio": 0.99000000953674316, 2024-06-03T05:25:50.830 INFO:tasks.workunit.client.0.smithi088.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-06-03T05:25:50.830 INFO:tasks.workunit.client.0.smithi088.stdout: "nearfull_ratio": 0.85000002384185791, 2024-06-03T05:25:50.830 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_snapshot": "", 2024-06-03T05:25:50.831 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_max": 1, 2024-06-03T05:25:50.831 INFO:tasks.workunit.client.0.smithi088.stdout: "max_osd": 1, 2024-06-03T05:25:50.831 INFO:tasks.workunit.client.0.smithi088.stdout: "require_min_compat_client": "luminous", 2024-06-03T05:25:50.831 INFO:tasks.workunit.client.0.smithi088.stdout: "min_compat_client": "jewel", 2024-06-03T05:25:50.831 INFO:tasks.workunit.client.0.smithi088.stdout: "require_osd_release": "squid", 2024-06-03T05:25:50.831 INFO:tasks.workunit.client.0.smithi088.stdout: "allow_crimson": false, 2024-06-03T05:25:50.831 INFO:tasks.workunit.client.0.smithi088.stdout: "pools": [ 2024-06-03T05:25:50.831 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.831 INFO:tasks.workunit.client.0.smithi088.stdout: "pool": 1, 2024-06-03T05:25:50.831 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_name": "rbd", 2024-06-03T05:25:50.831 INFO:tasks.workunit.client.0.smithi088.stdout: "create_time": "2024-06-03T05:25:36.239471+0000", 2024-06-03T05:25:50.831 INFO:tasks.workunit.client.0.smithi088.stdout: "flags": 8193, 2024-06-03T05:25:50.831 INFO:tasks.workunit.client.0.smithi088.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-06-03T05:25:50.832 INFO:tasks.workunit.client.0.smithi088.stdout: "type": 1, 2024-06-03T05:25:50.832 INFO:tasks.workunit.client.0.smithi088.stdout: "size": 2, 2024-06-03T05:25:50.832 INFO:tasks.workunit.client.0.smithi088.stdout: "min_size": 1, 2024-06-03T05:25:50.832 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_rule": 0, 2024-06-03T05:25:50.832 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_count": 0, 2024-06-03T05:25:50.832 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_target": 0, 2024-06-03T05:25:50.832 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_barrier": 0, 2024-06-03T05:25:50.832 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-06-03T05:25:50.832 INFO:tasks.workunit.client.0.smithi088.stdout: "object_hash": 2, 2024-06-03T05:25:50.832 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_autoscale_mode": "off", 2024-06-03T05:25:50.833 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_num": 4, 2024-06-03T05:25:50.833 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_placement_num": 4, 2024-06-03T05:25:50.833 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_placement_num_target": 4, 2024-06-03T05:25:50.833 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_num_target": 4, 2024-06-03T05:25:50.833 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_num_pending": 4, 2024-06-03T05:25:50.834 INFO:tasks.workunit.client.0.smithi088.stdout: "last_pg_merge_meta": { 2024-06-03T05:25:50.834 INFO:tasks.workunit.client.0.smithi088.stdout: "source_pgid": "0.0", 2024-06-03T05:25:50.834 INFO:tasks.workunit.client.0.smithi088.stdout: "ready_epoch": 0, 2024-06-03T05:25:50.834 INFO:tasks.workunit.client.0.smithi088.stdout: "last_epoch_started": 0, 2024-06-03T05:25:50.834 INFO:tasks.workunit.client.0.smithi088.stdout: "last_epoch_clean": 0, 2024-06-03T05:25:50.834 INFO:tasks.workunit.client.0.smithi088.stdout: "source_version": "0'0", 2024-06-03T05:25:50.834 INFO:tasks.workunit.client.0.smithi088.stdout: "target_version": "0'0" 2024-06-03T05:25:50.834 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.834 INFO:tasks.workunit.client.0.smithi088.stdout: "last_change": "12", 2024-06-03T05:25:50.834 INFO:tasks.workunit.client.0.smithi088.stdout: "last_force_op_resend": "0", 2024-06-03T05:25:50.834 INFO:tasks.workunit.client.0.smithi088.stdout: "last_force_op_resend_prenautilus": "0", 2024-06-03T05:25:50.835 INFO:tasks.workunit.client.0.smithi088.stdout: "last_force_op_resend_preluminous": "0", 2024-06-03T05:25:50.835 INFO:tasks.workunit.client.0.smithi088.stdout: "auid": 0, 2024-06-03T05:25:50.835 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_mode": "selfmanaged", 2024-06-03T05:25:50.835 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_seq": 2, 2024-06-03T05:25:50.835 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_epoch": 12, 2024-06-03T05:25:50.835 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_snaps": [], 2024-06-03T05:25:50.835 INFO:tasks.workunit.client.0.smithi088.stdout: "removed_snaps": "[]", 2024-06-03T05:25:50.835 INFO:tasks.workunit.client.0.smithi088.stdout: "quota_max_bytes": 0, 2024-06-03T05:25:50.835 INFO:tasks.workunit.client.0.smithi088.stdout: "quota_max_objects": 0, 2024-06-03T05:25:50.835 INFO:tasks.workunit.client.0.smithi088.stdout: "tiers": [], 2024-06-03T05:25:50.835 INFO:tasks.workunit.client.0.smithi088.stdout: "tier_of": -1, 2024-06-03T05:25:50.835 INFO:tasks.workunit.client.0.smithi088.stdout: "read_tier": -1, 2024-06-03T05:25:50.835 INFO:tasks.workunit.client.0.smithi088.stdout: "write_tier": -1, 2024-06-03T05:25:50.836 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_mode": "none", 2024-06-03T05:25:50.836 INFO:tasks.workunit.client.0.smithi088.stdout: "target_max_bytes": 0, 2024-06-03T05:25:50.836 INFO:tasks.workunit.client.0.smithi088.stdout: "target_max_objects": 0, 2024-06-03T05:25:50.836 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-06-03T05:25:50.836 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-06-03T05:25:50.836 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_target_full_ratio_micro": 800000, 2024-06-03T05:25:50.836 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_min_flush_age": 0, 2024-06-03T05:25:50.836 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_min_evict_age": 0, 2024-06-03T05:25:50.836 INFO:tasks.workunit.client.0.smithi088.stdout: "erasure_code_profile": "", 2024-06-03T05:25:50.836 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_params": { 2024-06-03T05:25:50.836 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "none" 2024-06-03T05:25:50.836 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.836 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_period": 0, 2024-06-03T05:25:50.836 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_count": 0, 2024-06-03T05:25:50.837 INFO:tasks.workunit.client.0.smithi088.stdout: "use_gmt_hitset": true, 2024-06-03T05:25:50.837 INFO:tasks.workunit.client.0.smithi088.stdout: "min_read_recency_for_promote": 0, 2024-06-03T05:25:50.837 INFO:tasks.workunit.client.0.smithi088.stdout: "min_write_recency_for_promote": 0, 2024-06-03T05:25:50.837 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_grade_decay_rate": 0, 2024-06-03T05:25:50.837 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_search_last_n": 0, 2024-06-03T05:25:50.837 INFO:tasks.workunit.client.0.smithi088.stdout: "grade_table": [], 2024-06-03T05:25:50.837 INFO:tasks.workunit.client.0.smithi088.stdout: "stripe_width": 0, 2024-06-03T05:25:50.837 INFO:tasks.workunit.client.0.smithi088.stdout: "expected_num_objects": 0, 2024-06-03T05:25:50.837 INFO:tasks.workunit.client.0.smithi088.stdout: "fast_read": false, 2024-06-03T05:25:50.837 INFO:tasks.workunit.client.0.smithi088.stdout: "options": {}, 2024-06-03T05:25:50.837 INFO:tasks.workunit.client.0.smithi088.stdout: "application_metadata": { 2024-06-03T05:25:50.837 INFO:tasks.workunit.client.0.smithi088.stdout: "rbd": {} 2024-06-03T05:25:50.837 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.838 INFO:tasks.workunit.client.0.smithi088.stdout: "read_balance": { 2024-06-03T05:25:50.838 INFO:tasks.workunit.client.0.smithi088.stdout: "score_type": "Fair distribution", 2024-06-03T05:25:50.838 INFO:tasks.workunit.client.0.smithi088.stdout: "score_acting": 1, 2024-06-03T05:25:50.838 INFO:tasks.workunit.client.0.smithi088.stdout: "score_stable": 1, 2024-06-03T05:25:50.838 INFO:tasks.workunit.client.0.smithi088.stdout: "optimal_score": 1, 2024-06-03T05:25:50.838 INFO:tasks.workunit.client.0.smithi088.stdout: "raw_score_acting": 1, 2024-06-03T05:25:50.838 INFO:tasks.workunit.client.0.smithi088.stdout: "raw_score_stable": 1, 2024-06-03T05:25:50.838 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_affinity_weighted": 1, 2024-06-03T05:25:50.839 INFO:tasks.workunit.client.0.smithi088.stdout: "average_primary_affinity": 1, 2024-06-03T05:25:50.839 INFO:tasks.workunit.client.0.smithi088.stdout: "average_primary_affinity_weighted": 1 2024-06-03T05:25:50.839 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:25:50.839 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:25:50.839 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-06-03T05:25:50.839 INFO:tasks.workunit.client.0.smithi088.stdout: "osds": [ 2024-06-03T05:25:50.839 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.839 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 0, 2024-06-03T05:25:50.839 INFO:tasks.workunit.client.0.smithi088.stdout: "uuid": "ff814cf5-edfb-4904-bc3c-e95d281ff14b", 2024-06-03T05:25:50.839 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 1, 2024-06-03T05:25:50.839 INFO:tasks.workunit.client.0.smithi088.stdout: "in": 1, 2024-06-03T05:25:50.839 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 1, 2024-06-03T05:25:50.839 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_affinity": 1, 2024-06-03T05:25:50.839 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_begin": 0, 2024-06-03T05:25:50.839 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_end": 0, 2024-06-03T05:25:50.840 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 5, 2024-06-03T05:25:50.840 INFO:tasks.workunit.client.0.smithi088.stdout: "up_thru": 7, 2024-06-03T05:25:50.840 INFO:tasks.workunit.client.0.smithi088.stdout: "down_at": 0, 2024-06-03T05:25:50.840 INFO:tasks.workunit.client.0.smithi088.stdout: "lost_at": 0, 2024-06-03T05:25:50.840 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addrs": { 2024-06-03T05:25:50.840 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-06-03T05:25:50.840 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.840 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-06-03T05:25:50.840 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6800", 2024-06-03T05:25:50.840 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2780887385 2024-06-03T05:25:50.840 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.840 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.840 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-06-03T05:25:50.840 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6801", 2024-06-03T05:25:50.840 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2780887385 2024-06-03T05:25:50.841 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:25:50.841 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-06-03T05:25:50.841 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.841 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addrs": { 2024-06-03T05:25:50.841 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-06-03T05:25:50.841 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.841 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-06-03T05:25:50.841 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6802", 2024-06-03T05:25:50.841 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2780887385 2024-06-03T05:25:50.841 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.841 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.841 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-06-03T05:25:50.841 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6803", 2024-06-03T05:25:50.841 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2780887385 2024-06-03T05:25:50.842 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:25:50.842 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-06-03T05:25:50.842 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.842 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addrs": { 2024-06-03T05:25:50.842 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-06-03T05:25:50.842 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.842 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-06-03T05:25:50.842 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6806", 2024-06-03T05:25:50.842 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2780887385 2024-06-03T05:25:50.842 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.842 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.842 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-06-03T05:25:50.843 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6807", 2024-06-03T05:25:50.843 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2780887385 2024-06-03T05:25:50.843 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:25:50.843 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-06-03T05:25:50.843 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.843 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addrs": { 2024-06-03T05:25:50.843 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-06-03T05:25:50.843 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.843 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-06-03T05:25:50.843 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6804", 2024-06-03T05:25:50.843 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2780887385 2024-06-03T05:25:50.844 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.844 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.844 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-06-03T05:25:50.844 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6805", 2024-06-03T05:25:50.844 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2780887385 2024-06-03T05:25:50.844 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:25:50.844 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-06-03T05:25:50.844 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.844 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addr": "127.0.0.1:6801/2780887385", 2024-06-03T05:25:50.844 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addr": "127.0.0.1:6803/2780887385", 2024-06-03T05:25:50.844 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addr": "127.0.0.1:6807/2780887385", 2024-06-03T05:25:50.844 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addr": "127.0.0.1:6805/2780887385", 2024-06-03T05:25:50.844 INFO:tasks.workunit.client.0.smithi088.stdout: "state": [ 2024-06-03T05:25:50.845 INFO:tasks.workunit.client.0.smithi088.stdout: "exists", 2024-06-03T05:25:50.845 INFO:tasks.workunit.client.0.smithi088.stdout: "up" 2024-06-03T05:25:50.845 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-06-03T05:25:50.845 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:25:50.845 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-06-03T05:25:50.845 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_xinfo": [ 2024-06-03T05:25:50.845 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.845 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 0, 2024-06-03T05:25:50.845 INFO:tasks.workunit.client.0.smithi088.stdout: "down_stamp": "0.000000", 2024-06-03T05:25:50.845 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_probability": 0, 2024-06-03T05:25:50.845 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_interval": 0, 2024-06-03T05:25:50.845 INFO:tasks.workunit.client.0.smithi088.stdout: "features": 4540701547738038271, 2024-06-03T05:25:50.845 INFO:tasks.workunit.client.0.smithi088.stdout: "old_weight": 0, 2024-06-03T05:25:50.845 INFO:tasks.workunit.client.0.smithi088.stdout: "last_purged_snaps_scrub": "2024-06-03T05:25:34.337636+0000", 2024-06-03T05:25:50.846 INFO:tasks.workunit.client.0.smithi088.stdout: "dead_epoch": 0 2024-06-03T05:25:50.846 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:25:50.846 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-06-03T05:25:50.846 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_upmap": [], 2024-06-03T05:25:50.846 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_upmap_items": [], 2024-06-03T05:25:50.846 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_upmap_primaries": [], 2024-06-03T05:25:50.846 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_temp": [], 2024-06-03T05:25:50.846 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_temp": [], 2024-06-03T05:25:50.846 INFO:tasks.workunit.client.0.smithi088.stdout: "blocklist": {}, 2024-06-03T05:25:50.846 INFO:tasks.workunit.client.0.smithi088.stdout: "range_blocklist": {}, 2024-06-03T05:25:50.846 INFO:tasks.workunit.client.0.smithi088.stdout: "erasure_code_profiles": { 2024-06-03T05:25:50.846 INFO:tasks.workunit.client.0.smithi088.stdout: "default": { 2024-06-03T05:25:50.846 INFO:tasks.workunit.client.0.smithi088.stdout: "k": "2", 2024-06-03T05:25:50.847 INFO:tasks.workunit.client.0.smithi088.stdout: "m": "2", 2024-06-03T05:25:50.847 INFO:tasks.workunit.client.0.smithi088.stdout: "plugin": "jerasure", 2024-06-03T05:25:50.847 INFO:tasks.workunit.client.0.smithi088.stdout: "technique": "reed_sol_van" 2024-06-03T05:25:50.847 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:25:50.847 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.847 INFO:tasks.workunit.client.0.smithi088.stdout: "removed_snaps_queue": [ 2024-06-03T05:25:50.847 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.847 INFO:tasks.workunit.client.0.smithi088.stdout: "pool": 1, 2024-06-03T05:25:50.847 INFO:tasks.workunit.client.0.smithi088.stdout: "snaps": [ 2024-06-03T05:25:50.847 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.847 INFO:tasks.workunit.client.0.smithi088.stdout: "begin": 2, 2024-06-03T05:25:50.847 INFO:tasks.workunit.client.0.smithi088.stdout: "length": 1 2024-06-03T05:25:50.847 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:25:50.847 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-06-03T05:25:50.848 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:25:50.848 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-06-03T05:25:50.848 INFO:tasks.workunit.client.0.smithi088.stdout: "new_removed_snaps": [ 2024-06-03T05:25:50.848 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.848 INFO:tasks.workunit.client.0.smithi088.stdout: "pool": 1, 2024-06-03T05:25:50.848 INFO:tasks.workunit.client.0.smithi088.stdout: "snaps": [ 2024-06-03T05:25:50.848 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.848 INFO:tasks.workunit.client.0.smithi088.stdout: "begin": 2, 2024-06-03T05:25:50.848 INFO:tasks.workunit.client.0.smithi088.stdout: "length": 1 2024-06-03T05:25:50.848 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:25:50.848 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-06-03T05:25:50.848 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:25:50.848 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-06-03T05:25:50.848 INFO:tasks.workunit.client.0.smithi088.stdout: "new_purged_snaps": [], 2024-06-03T05:25:50.848 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_node_flags": {}, 2024-06-03T05:25:50.849 INFO:tasks.workunit.client.0.smithi088.stdout: "device_class_flags": {}, 2024-06-03T05:25:50.849 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_mode": { 2024-06-03T05:25:50.849 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_mode_enabled": false, 2024-06-03T05:25:50.849 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_bucket_count": 0, 2024-06-03T05:25:50.849 INFO:tasks.workunit.client.0.smithi088.stdout: "degraded_stretch_mode": 0, 2024-06-03T05:25:50.849 INFO:tasks.workunit.client.0.smithi088.stdout: "recovering_stretch_mode": 0, 2024-06-03T05:25:50.849 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_mode_bucket": 0 2024-06-03T05:25:50.849 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:25:50.849 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.849 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_metadata": [ 2024-06-03T05:25:50.849 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.849 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-06-03T05:25:50.849 INFO:tasks.workunit.client.0.smithi088.stdout: "arch": "x86_64", 2024-06-03T05:25:50.849 INFO:tasks.workunit.client.0.smithi088.stdout: "back_addr": "[v2:127.0.0.1:6802/2780887385,v1:127.0.0.1:6803/2780887385]", 2024-06-03T05:25:50.850 INFO:tasks.workunit.client.0.smithi088.stdout: "back_iface": "", 2024-06-03T05:25:50.850 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs": "1", 2024-06-03T05:25:50.850 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_db": "0", 2024-06-03T05:25:50.850 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_wal": "0", 2024-06-03T05:25:50.850 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_single_shared_device": "1", 2024-06-03T05:25:50.850 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_allocation_from_file": "0", 2024-06-03T05:25:50.850 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_access_mode": "file", 2024-06-03T05:25:50.850 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_block_size": "4096", 2024-06-03T05:25:50.850 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_devices": "sda", 2024-06-03T05:25:50.850 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-06-03T05:25:50.850 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-06-03T05:25:50.850 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_path": "td/ceph-helpers/0/block", 2024-06-03T05:25:50.850 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_rotational": "1", 2024-06-03T05:25:50.850 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_size": "107374182400", 2024-06-03T05:25:50.851 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_support_discard": "0", 2024-06-03T05:25:50.851 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_type": "hdd", 2024-06-03T05:25:50.851 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_min_alloc_size": "4096", 2024-06-03T05:25:50.851 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_release": "squid", 2024-06-03T05:25:50.851 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version": "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)", 2024-06-03T05:25:50.851 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_short": "19.0.0-2946-g3d89accf", 2024-06-03T05:25:50.851 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_when_created": "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)", 2024-06-03T05:25:50.851 INFO:tasks.workunit.client.0.smithi088.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-06-03T05:25:50.851 INFO:tasks.workunit.client.0.smithi088.stdout: "created_at": "2024-06-03T05:25:07.857176Z", 2024-06-03T05:25:50.851 INFO:tasks.workunit.client.0.smithi088.stdout: "default_device_class": "hdd", 2024-06-03T05:25:50.851 INFO:tasks.workunit.client.0.smithi088.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHZ6", 2024-06-03T05:25:50.851 INFO:tasks.workunit.client.0.smithi088.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-06-03T05:25:50.851 INFO:tasks.workunit.client.0.smithi088.stdout: "devices": "sda", 2024-06-03T05:25:50.851 INFO:tasks.workunit.client.0.smithi088.stdout: "distro": "ubuntu", 2024-06-03T05:25:50.852 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-06-03T05:25:50.852 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_version": "22.04", 2024-06-03T05:25:50.852 INFO:tasks.workunit.client.0.smithi088.stdout: "front_addr": "[v2:127.0.0.1:6800/2780887385,v1:127.0.0.1:6801/2780887385]", 2024-06-03T05:25:50.852 INFO:tasks.workunit.client.0.smithi088.stdout: "front_iface": "", 2024-06-03T05:25:50.852 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_back_addr": "[v2:127.0.0.1:6806/2780887385,v1:127.0.0.1:6807/2780887385]", 2024-06-03T05:25:50.852 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_front_addr": "[v2:127.0.0.1:6804/2780887385,v1:127.0.0.1:6805/2780887385]", 2024-06-03T05:25:50.852 INFO:tasks.workunit.client.0.smithi088.stdout: "hostname": "smithi088", 2024-06-03T05:25:50.852 INFO:tasks.workunit.client.0.smithi088.stdout: "journal_rotational": "1", 2024-06-03T05:25:50.852 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_description": "#117-Ubuntu SMP Fri Apr 26 12:26:49 UTC 2024", 2024-06-03T05:25:50.852 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_version": "5.15.0-107-generic", 2024-06-03T05:25:50.852 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_swap_kb": "2097148", 2024-06-03T05:25:50.852 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_total_kb": "32746872", 2024-06-03T05:25:50.852 INFO:tasks.workunit.client.0.smithi088.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-06-03T05:25:50.852 INFO:tasks.workunit.client.0.smithi088.stdout: "objectstore_numa_unknown_devices": "sda", 2024-06-03T05:25:50.853 INFO:tasks.workunit.client.0.smithi088.stdout: "os": "Linux", 2024-06-03T05:25:50.853 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_data": "td/ceph-helpers/0", 2024-06-03T05:25:50.853 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_objectstore": "bluestore", 2024-06-03T05:25:50.853 INFO:tasks.workunit.client.0.smithi088.stdout: "osdspec_affinity": "", 2024-06-03T05:25:50.853 INFO:tasks.workunit.client.0.smithi088.stdout: "rotational": "1" 2024-06-03T05:25:50.853 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:25:50.853 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-06-03T05:25:50.853 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap_clean_epochs": { 2024-06-03T05:25:50.853 INFO:tasks.workunit.client.0.smithi088.stdout: "min_last_epoch_clean": 0, 2024-06-03T05:25:50.853 INFO:tasks.workunit.client.0.smithi088.stdout: "last_epoch_clean": { 2024-06-03T05:25:50.853 INFO:tasks.workunit.client.0.smithi088.stdout: "per_pool": [] 2024-06-03T05:25:50.853 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.853 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_epochs": [ 2024-06-03T05:25:50.853 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.854 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-06-03T05:25:50.854 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 5 2024-06-03T05:25:50.854 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:25:50.854 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-06-03T05:25:50.854 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.854 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap_first_committed": 1, 2024-06-03T05:25:50.854 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap_last_committed": 12, 2024-06-03T05:25:50.854 INFO:tasks.workunit.client.0.smithi088.stdout: "crushmap": { 2024-06-03T05:25:50.854 INFO:tasks.workunit.client.0.smithi088.stdout: "devices": [ 2024-06-03T05:25:50.854 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.854 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-06-03T05:25:50.854 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "osd.0", 2024-06-03T05:25:50.854 INFO:tasks.workunit.client.0.smithi088.stdout: "class": "hdd" 2024-06-03T05:25:50.855 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:25:50.855 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-06-03T05:25:50.855 INFO:tasks.workunit.client.0.smithi088.stdout: "types": [ 2024-06-03T05:25:50.855 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.855 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 0, 2024-06-03T05:25:50.855 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "osd" 2024-06-03T05:25:50.855 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.855 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.855 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 1, 2024-06-03T05:25:50.855 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "host" 2024-06-03T05:25:50.855 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.855 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.855 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 2, 2024-06-03T05:25:50.855 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "chassis" 2024-06-03T05:25:50.856 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.856 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.856 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 3, 2024-06-03T05:25:50.856 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "rack" 2024-06-03T05:25:50.856 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.856 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.856 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 4, 2024-06-03T05:25:50.856 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "row" 2024-06-03T05:25:50.856 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.856 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.856 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 5, 2024-06-03T05:25:50.856 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "pdu" 2024-06-03T05:25:50.856 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.856 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.857 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 6, 2024-06-03T05:25:50.857 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "pod" 2024-06-03T05:25:50.857 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.857 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.857 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 7, 2024-06-03T05:25:50.857 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "room" 2024-06-03T05:25:50.857 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.857 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.857 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 8, 2024-06-03T05:25:50.857 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "datacenter" 2024-06-03T05:25:50.857 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.857 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.857 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 9, 2024-06-03T05:25:50.858 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "zone" 2024-06-03T05:25:50.858 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.858 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.858 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 10, 2024-06-03T05:25:50.858 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "region" 2024-06-03T05:25:50.858 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.858 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.858 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 11, 2024-06-03T05:25:50.858 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "root" 2024-06-03T05:25:50.858 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:25:50.859 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-06-03T05:25:50.859 INFO:tasks.workunit.client.0.smithi088.stdout: "buckets": [ 2024-06-03T05:25:50.859 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.859 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -1, 2024-06-03T05:25:50.859 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "default", 2024-06-03T05:25:50.859 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 11, 2024-06-03T05:25:50.859 INFO:tasks.workunit.client.0.smithi088.stdout: "type_name": "root", 2024-06-03T05:25:50.859 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-06-03T05:25:50.859 INFO:tasks.workunit.client.0.smithi088.stdout: "alg": "straw2", 2024-06-03T05:25:50.859 INFO:tasks.workunit.client.0.smithi088.stdout: "hash": "rjenkins1", 2024-06-03T05:25:50.859 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-06-03T05:25:50.859 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.859 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -3, 2024-06-03T05:25:50.859 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-06-03T05:25:50.860 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 0 2024-06-03T05:25:50.860 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:25:50.860 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-06-03T05:25:50.860 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.860 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.860 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -2, 2024-06-03T05:25:50.860 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "default~hdd", 2024-06-03T05:25:50.860 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 11, 2024-06-03T05:25:50.860 INFO:tasks.workunit.client.0.smithi088.stdout: "type_name": "root", 2024-06-03T05:25:50.860 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-06-03T05:25:50.860 INFO:tasks.workunit.client.0.smithi088.stdout: "alg": "straw2", 2024-06-03T05:25:50.860 INFO:tasks.workunit.client.0.smithi088.stdout: "hash": "rjenkins1", 2024-06-03T05:25:50.860 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-06-03T05:25:50.860 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.861 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -4, 2024-06-03T05:25:50.861 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-06-03T05:25:50.861 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 0 2024-06-03T05:25:50.861 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:25:50.861 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-06-03T05:25:50.861 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.861 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.861 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -3, 2024-06-03T05:25:50.861 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "smithi088", 2024-06-03T05:25:50.861 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 1, 2024-06-03T05:25:50.861 INFO:tasks.workunit.client.0.smithi088.stdout: "type_name": "host", 2024-06-03T05:25:50.861 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-06-03T05:25:50.861 INFO:tasks.workunit.client.0.smithi088.stdout: "alg": "straw2", 2024-06-03T05:25:50.862 INFO:tasks.workunit.client.0.smithi088.stdout: "hash": "rjenkins1", 2024-06-03T05:25:50.862 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-06-03T05:25:50.862 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.862 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-06-03T05:25:50.862 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-06-03T05:25:50.862 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 0 2024-06-03T05:25:50.862 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:25:50.862 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-06-03T05:25:50.862 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.862 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.862 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -4, 2024-06-03T05:25:50.862 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "smithi088~hdd", 2024-06-03T05:25:50.862 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 1, 2024-06-03T05:25:50.863 INFO:tasks.workunit.client.0.smithi088.stdout: "type_name": "host", 2024-06-03T05:25:50.863 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-06-03T05:25:50.863 INFO:tasks.workunit.client.0.smithi088.stdout: "alg": "straw2", 2024-06-03T05:25:50.863 INFO:tasks.workunit.client.0.smithi088.stdout: "hash": "rjenkins1", 2024-06-03T05:25:50.863 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-06-03T05:25:50.863 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.863 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-06-03T05:25:50.863 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-06-03T05:25:50.863 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 0 2024-06-03T05:25:50.863 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:25:50.863 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-06-03T05:25:50.863 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:25:50.863 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-06-03T05:25:50.863 INFO:tasks.workunit.client.0.smithi088.stdout: "rules": [ 2024-06-03T05:25:50.863 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.864 INFO:tasks.workunit.client.0.smithi088.stdout: "rule_id": 0, 2024-06-03T05:25:50.864 INFO:tasks.workunit.client.0.smithi088.stdout: "rule_name": "replicated_rule", 2024-06-03T05:25:50.864 INFO:tasks.workunit.client.0.smithi088.stdout: "type": 1, 2024-06-03T05:25:50.864 INFO:tasks.workunit.client.0.smithi088.stdout: "steps": [ 2024-06-03T05:25:50.864 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.864 INFO:tasks.workunit.client.0.smithi088.stdout: "op": "take", 2024-06-03T05:25:50.864 INFO:tasks.workunit.client.0.smithi088.stdout: "item": -1, 2024-06-03T05:25:50.864 INFO:tasks.workunit.client.0.smithi088.stdout: "item_name": "default" 2024-06-03T05:25:50.864 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.864 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.864 INFO:tasks.workunit.client.0.smithi088.stdout: "op": "choose_firstn", 2024-06-03T05:25:50.864 INFO:tasks.workunit.client.0.smithi088.stdout: "num": 0, 2024-06-03T05:25:50.864 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "osd" 2024-06-03T05:25:50.865 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.865 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.865 INFO:tasks.workunit.client.0.smithi088.stdout: "op": "emit" 2024-06-03T05:25:50.865 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:25:50.865 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-06-03T05:25:50.865 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:25:50.865 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-06-03T05:25:50.865 INFO:tasks.workunit.client.0.smithi088.stdout: "tunables": { 2024-06-03T05:25:50.865 INFO:tasks.workunit.client.0.smithi088.stdout: "choose_local_tries": 0, 2024-06-03T05:25:50.865 INFO:tasks.workunit.client.0.smithi088.stdout: "choose_local_fallback_tries": 0, 2024-06-03T05:25:50.865 INFO:tasks.workunit.client.0.smithi088.stdout: "choose_total_tries": 50, 2024-06-03T05:25:50.865 INFO:tasks.workunit.client.0.smithi088.stdout: "chooseleaf_descend_once": 1, 2024-06-03T05:25:50.865 INFO:tasks.workunit.client.0.smithi088.stdout: "chooseleaf_vary_r": 1, 2024-06-03T05:25:50.865 INFO:tasks.workunit.client.0.smithi088.stdout: "chooseleaf_stable": 1, 2024-06-03T05:25:50.866 INFO:tasks.workunit.client.0.smithi088.stdout: "msr_descents": 100, 2024-06-03T05:25:50.866 INFO:tasks.workunit.client.0.smithi088.stdout: "msr_collision_tries": 100, 2024-06-03T05:25:50.866 INFO:tasks.workunit.client.0.smithi088.stdout: "straw_calc_version": 1, 2024-06-03T05:25:50.866 INFO:tasks.workunit.client.0.smithi088.stdout: "allowed_bucket_algs": 54, 2024-06-03T05:25:50.866 INFO:tasks.workunit.client.0.smithi088.stdout: "profile": "jewel", 2024-06-03T05:25:50.866 INFO:tasks.workunit.client.0.smithi088.stdout: "optimal_tunables": 1, 2024-06-03T05:25:50.866 INFO:tasks.workunit.client.0.smithi088.stdout: "legacy_tunables": 0, 2024-06-03T05:25:50.866 INFO:tasks.workunit.client.0.smithi088.stdout: "minimum_required_version": "jewel", 2024-06-03T05:25:50.866 INFO:tasks.workunit.client.0.smithi088.stdout: "require_feature_tunables": 1, 2024-06-03T05:25:50.866 INFO:tasks.workunit.client.0.smithi088.stdout: "require_feature_tunables2": 1, 2024-06-03T05:25:50.866 INFO:tasks.workunit.client.0.smithi088.stdout: "has_v2_rules": 0, 2024-06-03T05:25:50.866 INFO:tasks.workunit.client.0.smithi088.stdout: "require_feature_tunables3": 1, 2024-06-03T05:25:50.866 INFO:tasks.workunit.client.0.smithi088.stdout: "has_v3_rules": 0, 2024-06-03T05:25:50.867 INFO:tasks.workunit.client.0.smithi088.stdout: "has_v4_buckets": 1, 2024-06-03T05:25:50.867 INFO:tasks.workunit.client.0.smithi088.stdout: "require_feature_tunables5": 1, 2024-06-03T05:25:50.867 INFO:tasks.workunit.client.0.smithi088.stdout: "has_v5_rules": 0, 2024-06-03T05:25:50.867 INFO:tasks.workunit.client.0.smithi088.stdout: "has_msr_rules": 0 2024-06-03T05:25:50.867 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.867 INFO:tasks.workunit.client.0.smithi088.stdout: "choose_args": {} 2024-06-03T05:25:50.867 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.867 INFO:tasks.workunit.client.0.smithi088.stdout: "fsmap": { 2024-06-03T05:25:50.867 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 1, 2024-06-03T05:25:50.867 INFO:tasks.workunit.client.0.smithi088.stdout: "default_fscid": -1, 2024-06-03T05:25:50.867 INFO:tasks.workunit.client.0.smithi088.stdout: "compat": { 2024-06-03T05:25:50.867 INFO:tasks.workunit.client.0.smithi088.stdout: "compat": {}, 2024-06-03T05:25:50.867 INFO:tasks.workunit.client.0.smithi088.stdout: "ro_compat": {}, 2024-06-03T05:25:50.867 INFO:tasks.workunit.client.0.smithi088.stdout: "incompat": { 2024-06-03T05:25:50.868 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_1": "base v0.20", 2024-06-03T05:25:50.868 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_2": "client writeable ranges", 2024-06-03T05:25:50.868 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_3": "default file layouts on dirs", 2024-06-03T05:25:50.868 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_4": "dir inode in separate object", 2024-06-03T05:25:50.868 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_5": "mds uses versioned encoding", 2024-06-03T05:25:50.868 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_6": "dirfrag is stored in omap", 2024-06-03T05:25:50.868 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_8": "no anchor table", 2024-06-03T05:25:50.868 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_9": "file layout v2", 2024-06-03T05:25:50.868 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_10": "snaprealm v2", 2024-06-03T05:25:50.868 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_11": "minor log segments", 2024-06-03T05:25:50.868 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_12": "quiesce subvolumes" 2024-06-03T05:25:50.868 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:25:50.868 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.869 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_flags": { 2024-06-03T05:25:50.869 INFO:tasks.workunit.client.0.smithi088.stdout: "enable_multiple": true, 2024-06-03T05:25:50.869 INFO:tasks.workunit.client.0.smithi088.stdout: "ever_enabled_multiple": true 2024-06-03T05:25:50.869 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.869 INFO:tasks.workunit.client.0.smithi088.stdout: "standbys": [], 2024-06-03T05:25:50.869 INFO:tasks.workunit.client.0.smithi088.stdout: "filesystems": [] 2024-06-03T05:25:50.869 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.869 INFO:tasks.workunit.client.0.smithi088.stdout: "mdsmap_first_committed": 1, 2024-06-03T05:25:50.869 INFO:tasks.workunit.client.0.smithi088.stdout: "mdsmap_last_committed": 1, 2024-06-03T05:25:50.869 INFO:tasks.workunit.client.0.smithi088.stdout: "auth": { 2024-06-03T05:25:50.869 INFO:tasks.workunit.client.0.smithi088.stdout: "first_committed": 1, 2024-06-03T05:25:50.869 INFO:tasks.workunit.client.0.smithi088.stdout: "last_committed": 3, 2024-06-03T05:25:50.869 INFO:tasks.workunit.client.0.smithi088.stdout: "num_secrets": 1 2024-06-03T05:25:50.870 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.870 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg": 4, 2024-06-03T05:25:50.870 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg_active": 4, 2024-06-03T05:25:50.870 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg_unknown": 0, 2024-06-03T05:25:50.870 INFO:tasks.workunit.client.0.smithi088.stdout: "num_osd": 1, 2024-06-03T05:25:50.870 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_sum": { 2024-06-03T05:25:50.870 INFO:tasks.workunit.client.0.smithi088.stdout: "stat_sum": { 2024-06-03T05:25:50.870 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes": 19, 2024-06-03T05:25:50.870 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects": 1, 2024-06-03T05:25:50.870 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_clones": 0, 2024-06-03T05:25:50.870 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_copies": 2, 2024-06-03T05:25:50.870 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing_on_primary": 0, 2024-06-03T05:25:50.870 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing": 0, 2024-06-03T05:25:50.871 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_degraded": 1, 2024-06-03T05:25:50.871 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_misplaced": 0, 2024-06-03T05:25:50.871 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_unfound": 0, 2024-06-03T05:25:50.871 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_dirty": 1, 2024-06-03T05:25:50.871 INFO:tasks.workunit.client.0.smithi088.stdout: "num_whiteouts": 0, 2024-06-03T05:25:50.871 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read": 0, 2024-06-03T05:25:50.871 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read_kb": 0, 2024-06-03T05:25:50.871 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write": 2, 2024-06-03T05:25:50.871 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write_kb": 2, 2024-06-03T05:25:50.871 INFO:tasks.workunit.client.0.smithi088.stdout: "num_scrub_errors": 0, 2024-06-03T05:25:50.871 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shallow_scrub_errors": 0, 2024-06-03T05:25:50.871 INFO:tasks.workunit.client.0.smithi088.stdout: "num_deep_scrub_errors": 0, 2024-06-03T05:25:50.871 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_recovered": 0, 2024-06-03T05:25:50.872 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_recovered": 0, 2024-06-03T05:25:50.872 INFO:tasks.workunit.client.0.smithi088.stdout: "num_keys_recovered": 0, 2024-06-03T05:25:50.872 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_omap": 0, 2024-06-03T05:25:50.872 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_hit_set_archive": 0, 2024-06-03T05:25:50.872 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_hit_set_archive": 0, 2024-06-03T05:25:50.872 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush": 0, 2024-06-03T05:25:50.872 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_kb": 0, 2024-06-03T05:25:50.872 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict": 0, 2024-06-03T05:25:50.872 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_kb": 0, 2024-06-03T05:25:50.872 INFO:tasks.workunit.client.0.smithi088.stdout: "num_promote": 0, 2024-06-03T05:25:50.872 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_high": 0, 2024-06-03T05:25:50.872 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_low": 0, 2024-06-03T05:25:50.872 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_some": 0, 2024-06-03T05:25:50.872 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_full": 0, 2024-06-03T05:25:50.873 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_pinned": 0, 2024-06-03T05:25:50.873 INFO:tasks.workunit.client.0.smithi088.stdout: "num_legacy_snapsets": 0, 2024-06-03T05:25:50.873 INFO:tasks.workunit.client.0.smithi088.stdout: "num_large_omap_objects": 0, 2024-06-03T05:25:50.873 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_manifest": 0, 2024-06-03T05:25:50.873 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_bytes": 0, 2024-06-03T05:25:50.873 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_keys": 0, 2024-06-03T05:25:50.873 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_repaired": 0 2024-06-03T05:25:50.873 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.873 INFO:tasks.workunit.client.0.smithi088.stdout: "store_stats": { 2024-06-03T05:25:50.873 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 0, 2024-06-03T05:25:50.873 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 0, 2024-06-03T05:25:50.873 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-06-03T05:25:50.873 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 0, 2024-06-03T05:25:50.873 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 0, 2024-06-03T05:25:50.874 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-06-03T05:25:50.874 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-06-03T05:25:50.874 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-06-03T05:25:50.874 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 0, 2024-06-03T05:25:50.874 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 0 2024-06-03T05:25:50.874 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.874 INFO:tasks.workunit.client.0.smithi088.stdout: "log_size": 2, 2024-06-03T05:25:50.874 INFO:tasks.workunit.client.0.smithi088.stdout: "ondisk_log_size": 2, 2024-06-03T05:25:50.874 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 4, 2024-06-03T05:25:50.874 INFO:tasks.workunit.client.0.smithi088.stdout: "acting": 4, 2024-06-03T05:25:50.874 INFO:tasks.workunit.client.0.smithi088.stdout: "num_store_stats": 0 2024-06-03T05:25:50.874 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.874 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_sum": { 2024-06-03T05:25:50.874 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 0, 2024-06-03T05:25:50.875 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 0, 2024-06-03T05:25:50.875 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pgs": 4, 2024-06-03T05:25:50.875 INFO:tasks.workunit.client.0.smithi088.stdout: "num_osds": 1, 2024-06-03T05:25:50.875 INFO:tasks.workunit.client.0.smithi088.stdout: "num_per_pool_osds": 1, 2024-06-03T05:25:50.875 INFO:tasks.workunit.client.0.smithi088.stdout: "num_per_pool_omap_osds": 1, 2024-06-03T05:25:50.875 INFO:tasks.workunit.client.0.smithi088.stdout: "kb": 104857600, 2024-06-03T05:25:50.875 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used": 26948, 2024-06-03T05:25:50.875 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_data": 108, 2024-06-03T05:25:50.875 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_omap": 1, 2024-06-03T05:25:50.875 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_meta": 26814, 2024-06-03T05:25:50.876 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_avail": 104830652, 2024-06-03T05:25:50.876 INFO:tasks.workunit.client.0.smithi088.stdout: "statfs": { 2024-06-03T05:25:50.876 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 107374182400, 2024-06-03T05:25:50.876 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 107346587648, 2024-06-03T05:25:50.876 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-06-03T05:25:50.876 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 110592, 2024-06-03T05:25:50.876 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 22938, 2024-06-03T05:25:50.876 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-06-03T05:25:50.876 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-06-03T05:25:50.876 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-06-03T05:25:50.876 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 1595, 2024-06-03T05:25:50.876 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 27457989 2024-06-03T05:25:50.876 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.876 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_peers": [], 2024-06-03T05:25:50.877 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_trim_queue_len": 0, 2024-06-03T05:25:50.877 INFO:tasks.workunit.client.0.smithi088.stdout: "num_snap_trimming": 0, 2024-06-03T05:25:50.877 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shards_repaired": 0, 2024-06-03T05:25:50.877 INFO:tasks.workunit.client.0.smithi088.stdout: "op_queue_age_hist": { 2024-06-03T05:25:50.877 INFO:tasks.workunit.client.0.smithi088.stdout: "histogram": [], 2024-06-03T05:25:50.877 INFO:tasks.workunit.client.0.smithi088.stdout: "upper_bound": 1 2024-06-03T05:25:50.877 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.877 INFO:tasks.workunit.client.0.smithi088.stdout: "perf_stat": { 2024-06-03T05:25:50.877 INFO:tasks.workunit.client.0.smithi088.stdout: "commit_latency_ms": 0, 2024-06-03T05:25:50.877 INFO:tasks.workunit.client.0.smithi088.stdout: "apply_latency_ms": 0, 2024-06-03T05:25:50.877 INFO:tasks.workunit.client.0.smithi088.stdout: "commit_latency_ns": 0, 2024-06-03T05:25:50.877 INFO:tasks.workunit.client.0.smithi088.stdout: "apply_latency_ns": 0 2024-06-03T05:25:50.877 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.877 INFO:tasks.workunit.client.0.smithi088.stdout: "alerts": [], 2024-06-03T05:25:50.878 INFO:tasks.workunit.client.0.smithi088.stdout: "network_ping_times": [] 2024-06-03T05:25:50.878 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.878 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_sum_by_class": { 2024-06-03T05:25:50.878 INFO:tasks.workunit.client.0.smithi088.stdout: "hdd": { 2024-06-03T05:25:50.878 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 0, 2024-06-03T05:25:50.878 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 0, 2024-06-03T05:25:50.878 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pgs": 4, 2024-06-03T05:25:50.878 INFO:tasks.workunit.client.0.smithi088.stdout: "num_osds": 1, 2024-06-03T05:25:50.878 INFO:tasks.workunit.client.0.smithi088.stdout: "num_per_pool_osds": 1, 2024-06-03T05:25:50.878 INFO:tasks.workunit.client.0.smithi088.stdout: "num_per_pool_omap_osds": 1, 2024-06-03T05:25:50.878 INFO:tasks.workunit.client.0.smithi088.stdout: "kb": 104857600, 2024-06-03T05:25:50.878 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used": 26948, 2024-06-03T05:25:50.878 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_data": 108, 2024-06-03T05:25:50.878 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_omap": 1, 2024-06-03T05:25:50.879 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_meta": 26814, 2024-06-03T05:25:50.879 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_avail": 104830652, 2024-06-03T05:25:50.879 INFO:tasks.workunit.client.0.smithi088.stdout: "statfs": { 2024-06-03T05:25:50.879 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 107374182400, 2024-06-03T05:25:50.879 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 107346587648, 2024-06-03T05:25:50.879 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-06-03T05:25:50.879 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 110592, 2024-06-03T05:25:50.879 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 22938, 2024-06-03T05:25:50.879 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-06-03T05:25:50.879 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-06-03T05:25:50.879 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-06-03T05:25:50.879 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 1595, 2024-06-03T05:25:50.879 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 27457989 2024-06-03T05:25:50.879 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.880 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_peers": [], 2024-06-03T05:25:50.880 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_trim_queue_len": 0, 2024-06-03T05:25:50.880 INFO:tasks.workunit.client.0.smithi088.stdout: "num_snap_trimming": 0, 2024-06-03T05:25:50.880 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shards_repaired": 0, 2024-06-03T05:25:50.880 INFO:tasks.workunit.client.0.smithi088.stdout: "op_queue_age_hist": { 2024-06-03T05:25:50.880 INFO:tasks.workunit.client.0.smithi088.stdout: "histogram": [], 2024-06-03T05:25:50.880 INFO:tasks.workunit.client.0.smithi088.stdout: "upper_bound": 1 2024-06-03T05:25:50.880 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.880 INFO:tasks.workunit.client.0.smithi088.stdout: "perf_stat": { 2024-06-03T05:25:50.880 INFO:tasks.workunit.client.0.smithi088.stdout: "commit_latency_ms": 0, 2024-06-03T05:25:50.880 INFO:tasks.workunit.client.0.smithi088.stdout: "apply_latency_ms": 0, 2024-06-03T05:25:50.880 INFO:tasks.workunit.client.0.smithi088.stdout: "commit_latency_ns": 0, 2024-06-03T05:25:50.880 INFO:tasks.workunit.client.0.smithi088.stdout: "apply_latency_ns": 0 2024-06-03T05:25:50.880 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.881 INFO:tasks.workunit.client.0.smithi088.stdout: "alerts": [], 2024-06-03T05:25:50.881 INFO:tasks.workunit.client.0.smithi088.stdout: "network_ping_times": [] 2024-06-03T05:25:50.881 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:25:50.881 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.881 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_stats": [ 2024-06-03T05:25:50.881 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.881 INFO:tasks.workunit.client.0.smithi088.stdout: "poolid": 1, 2024-06-03T05:25:50.881 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg": 4, 2024-06-03T05:25:50.881 INFO:tasks.workunit.client.0.smithi088.stdout: "stat_sum": { 2024-06-03T05:25:50.881 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes": 19, 2024-06-03T05:25:50.881 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects": 1, 2024-06-03T05:25:50.881 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_clones": 0, 2024-06-03T05:25:50.881 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_copies": 2, 2024-06-03T05:25:50.881 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing_on_primary": 0, 2024-06-03T05:25:50.882 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing": 0, 2024-06-03T05:25:50.882 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_degraded": 1, 2024-06-03T05:25:50.882 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_misplaced": 0, 2024-06-03T05:25:50.882 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_unfound": 0, 2024-06-03T05:25:50.882 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_dirty": 1, 2024-06-03T05:25:50.882 INFO:tasks.workunit.client.0.smithi088.stdout: "num_whiteouts": 0, 2024-06-03T05:25:50.882 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read": 0, 2024-06-03T05:25:50.882 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read_kb": 0, 2024-06-03T05:25:50.882 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write": 2, 2024-06-03T05:25:50.882 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write_kb": 2, 2024-06-03T05:25:50.882 INFO:tasks.workunit.client.0.smithi088.stdout: "num_scrub_errors": 0, 2024-06-03T05:25:50.882 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shallow_scrub_errors": 0, 2024-06-03T05:25:50.882 INFO:tasks.workunit.client.0.smithi088.stdout: "num_deep_scrub_errors": 0, 2024-06-03T05:25:50.882 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_recovered": 0, 2024-06-03T05:25:50.883 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_recovered": 0, 2024-06-03T05:25:50.883 INFO:tasks.workunit.client.0.smithi088.stdout: "num_keys_recovered": 0, 2024-06-03T05:25:50.883 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_omap": 0, 2024-06-03T05:25:50.883 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_hit_set_archive": 0, 2024-06-03T05:25:50.883 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_hit_set_archive": 0, 2024-06-03T05:25:50.883 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush": 0, 2024-06-03T05:25:50.883 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_kb": 0, 2024-06-03T05:25:50.883 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict": 0, 2024-06-03T05:25:50.883 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_kb": 0, 2024-06-03T05:25:50.883 INFO:tasks.workunit.client.0.smithi088.stdout: "num_promote": 0, 2024-06-03T05:25:50.883 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_high": 0, 2024-06-03T05:25:50.883 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_low": 0, 2024-06-03T05:25:50.883 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_some": 0, 2024-06-03T05:25:50.883 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_full": 0, 2024-06-03T05:25:50.884 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_pinned": 0, 2024-06-03T05:25:50.884 INFO:tasks.workunit.client.0.smithi088.stdout: "num_legacy_snapsets": 0, 2024-06-03T05:25:50.884 INFO:tasks.workunit.client.0.smithi088.stdout: "num_large_omap_objects": 0, 2024-06-03T05:25:50.884 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_manifest": 0, 2024-06-03T05:25:50.884 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_bytes": 0, 2024-06-03T05:25:50.884 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_keys": 0, 2024-06-03T05:25:50.884 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_repaired": 0 2024-06-03T05:25:50.884 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.884 INFO:tasks.workunit.client.0.smithi088.stdout: "store_stats": { 2024-06-03T05:25:50.884 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 0, 2024-06-03T05:25:50.884 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 0, 2024-06-03T05:25:50.884 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-06-03T05:25:50.884 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 4096, 2024-06-03T05:25:50.884 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 19, 2024-06-03T05:25:50.885 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-06-03T05:25:50.885 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-06-03T05:25:50.885 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-06-03T05:25:50.885 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 0, 2024-06-03T05:25:50.885 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 0 2024-06-03T05:25:50.885 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.885 INFO:tasks.workunit.client.0.smithi088.stdout: "log_size": 2, 2024-06-03T05:25:50.885 INFO:tasks.workunit.client.0.smithi088.stdout: "ondisk_log_size": 2, 2024-06-03T05:25:50.885 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 4, 2024-06-03T05:25:50.885 INFO:tasks.workunit.client.0.smithi088.stdout: "acting": 4, 2024-06-03T05:25:50.885 INFO:tasks.workunit.client.0.smithi088.stdout: "num_store_stats": 1 2024-06-03T05:25:50.885 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:25:50.885 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-06-03T05:25:50.885 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_stats": [ 2024-06-03T05:25:50.885 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.886 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 0, 2024-06-03T05:25:50.886 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 21474836482 2024-06-03T05:25:50.886 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:25:50.886 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-06-03T05:25:50.886 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg_by_state": [ 2024-06-03T05:25:50.886 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.886 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "active+undersized+degraded", 2024-06-03T05:25:50.886 INFO:tasks.workunit.client.0.smithi088.stdout: "num": 1 2024-06-03T05:25:50.886 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.886 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.886 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "active+undersized", 2024-06-03T05:25:50.886 INFO:tasks.workunit.client.0.smithi088.stdout: "num": 3 2024-06-03T05:25:50.886 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:25:50.887 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-06-03T05:25:50.887 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg_by_osd": [ 2024-06-03T05:25:50.887 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.887 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 0, 2024-06-03T05:25:50.887 INFO:tasks.workunit.client.0.smithi088.stdout: "num_primary_pg": 4, 2024-06-03T05:25:50.887 INFO:tasks.workunit.client.0.smithi088.stdout: "num_acting_pg": 4, 2024-06-03T05:25:50.887 INFO:tasks.workunit.client.0.smithi088.stdout: "num_up_not_acting_pg": 0 2024-06-03T05:25:50.887 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:25:50.887 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-06-03T05:25:50.887 INFO:tasks.workunit.client.0.smithi088.stdout: "purged_snaps": [ 2024-06-03T05:25:50.887 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:25:50.887 INFO:tasks.workunit.client.0.smithi088.stdout: "pool": 1, 2024-06-03T05:25:50.887 INFO:tasks.workunit.client.0.smithi088.stdout: "purged_snaps": {} 2024-06-03T05:25:50.887 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:25:50.888 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-06-03T05:25:50.888 INFO:tasks.workunit.client.0.smithi088.stdout: "servicemap": { 2024-06-03T05:25:50.888 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 1, 2024-06-03T05:25:50.888 INFO:tasks.workunit.client.0.smithi088.stdout: "modified": "2024-06-03T05:25:01.438111+0000", 2024-06-03T05:25:50.888 INFO:tasks.workunit.client.0.smithi088.stdout: "services": {} 2024-06-03T05:25:50.888 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:25:50.888 INFO:tasks.workunit.client.0.smithi088.stdout: "mgrstat_first_committed": 1, 2024-06-03T05:25:50.888 INFO:tasks.workunit.client.0.smithi088.stdout: "mgrstat_last_committed": 7, 2024-06-03T05:25:50.888 INFO:tasks.workunit.client.0.smithi088.stdout: "logm_first_committed": 1, 2024-06-03T05:25:50.888 INFO:tasks.workunit.client.0.smithi088.stdout: "logm_last_committed": 78, 2024-06-03T05:25:50.888 INFO:tasks.workunit.client.0.smithi088.stdout: "paxos": { 2024-06-03T05:25:50.888 INFO:tasks.workunit.client.0.smithi088.stdout: "first_committed": 1, 2024-06-03T05:25:50.888 INFO:tasks.workunit.client.0.smithi088.stdout: "last_committed": 104, 2024-06-03T05:25:50.888 INFO:tasks.workunit.client.0.smithi088.stdout: "last_pn": 0, 2024-06-03T05:25:50.888 INFO:tasks.workunit.client.0.smithi088.stdout: "accepted_pn": 0 2024-06-03T05:25:50.889 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:25:50.889 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-06-03T05:25:50.891 INFO:tasks.workunit.client.0.smithi088.stderr:report 1210503162 2024-06-03T05:25:50.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-06-03T05:25:50.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1685: test_wait_for_clean: run_osd td/ceph-helpers 1 2024-06-03T05:25:50.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:25:50.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:25:50.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-03T05:25:50.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:25:50.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-06-03T05:25:50.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:25:50.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:25:50.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:25:50.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:25:50.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-06-03T05:25:50.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-06-03T05:25:50.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:25:50.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:25:50.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:25:50.893 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:25:50.893 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:25:50.893 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:25:50.893 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:25:50.893 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:25:50.893 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:25:50.893 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:25:50.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:25:50.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:25:50.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:25:50.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:25:50.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:25:50.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:25:50.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:25:50.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:25:50.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:25:50.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:25:50.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:25:50.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:25:50.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-06-03T05:25:50.894 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:25:50.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7dae1f32-821a-4f5d-a555-819a510aee61 2024-06-03T05:25:50.895 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 7dae1f32-821a-4f5d-a555-819a510aee61 2024-06-03T05:25:50.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 7dae1f32-821a-4f5d-a555-819a510aee61' 2024-06-03T05:25:50.895 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:25:50.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDeU11mcG17MxAApaM4/ZNva9/inBZ5kQSKpg== 2024-06-03T05:25:50.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDeU11mcG17MxAApaM4/ZNva9/inBZ5kQSKpg=="}' 2024-06-03T05:25:50.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7dae1f32-821a-4f5d-a555-819a510aee61 -i td/ceph-helpers/1/new.json 2024-06-03T05:25:51.330 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:25:51.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-06-03T05:25:51.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDeU11mcG17MxAApaM4/ZNva9/inBZ5kQSKpg== --osd-uuid 7dae1f32-821a-4f5d-a555-819a510aee61 2024-06-03T05:25:51.376 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:25:51.371+0000 7f208862d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:25:51.422 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:25:51.419+0000 7f208862d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:25:51.423 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:25:51.419+0000 7f208862d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:25:51.424 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:25:51.419+0000 7f208862d7c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-03T05:25:51.424 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:25:51.419+0000 7f208862d7c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-03T05:25:51.424 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:25:51.419+0000 7f208862d7c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-03T05:25:51.461 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:25:51.459+0000 7f208862d7c0 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-06-03T05:25:57.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-06-03T05:25:57.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:25:57.990 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-06-03T05:25:57.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-03T05:25:57.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:25:58.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-03T05:25:58.554 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-06-03T05:25:58.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:25:58.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:25:58.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:25:58.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:25:58.579 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:25:58.575+0000 7fee77df47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:25:58.585 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:25:58.579+0000 7fee77df47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:25:58.586 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:25:58.583+0000 7fee77df47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:25:59.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-03T05:25:59.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:25:59.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T05:25:59.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:25:59.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:25:59.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:25:59.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:25:59.015 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:25:59.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:25:59.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:25:59.139 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:25:59.135+0000 7fee77df47c0 -1 Falling back to public interface 2024-06-03T05:25:59.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:26:00.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:26:00.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:26:00.419 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:26:00.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:26:00.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:26:00.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:26:00.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:26:01.767 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:26:01.763+0000 7fee77df47c0 -1 osd.1 0 log_to_monitors true 2024-06-03T05:26:01.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:26:01.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:26:01.822 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:26:01.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:26:01.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:26:01.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:26:02.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:26:03.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:26:03.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:26:03.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:26:03.489 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:26:03.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:26:03.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:26:03.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:26:04.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:26:04.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:26:04.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:26:04.970 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:26:04.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:26:04.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:26:05.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:26:06.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:26:06.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:26:06.477 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:26:06.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:26:06.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:26:06.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:26:07.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:26:08.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:26:08.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:26:08.304 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:26:08.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:26:08.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:26:08.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:26:08.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:26:09.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:26:09.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:26:09.853 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:26:09.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:26:09.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:26:09.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:26:10.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:26:11.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:26:11.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:26:11.269 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:26:11.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:26:11.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:26:11.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:26:11.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:26:12.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:26:12.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:26:12.818 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:26:12.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:26:12.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:26:12.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:26:13.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:26:14.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:26:14.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:26:14.349 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:26:14.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:26:14.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:26:14.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:26:14.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:26:15.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:26:15.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:26:15.925 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:26:15.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:26:15.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:26:15.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:26:16.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:26:17.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:26:17.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:26:17.330 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:26:17.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:26:17.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:26:17.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:26:18.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:26:19.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:26:19.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:26:19.426 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:26:19.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:26:19.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:26:19.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:26:20.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:26:21.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:26:21.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:26:21.300 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:26:21.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:26:21.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:26:21.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:26:22.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:26:23.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:26:23.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:26:23.267 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:26:23.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:26:23.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:26:23.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:26:24.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:26:25.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:26:25.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:26:25.420 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:26:25.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:26:25.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:26:25.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:26:26.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:26:27.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:26:27.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:26:27.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:26:27.604 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:26:27.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:26:27.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:26:28.269 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 16 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3138443500,v1:127.0.0.1:6811/3138443500] [v2:127.0.0.1:6812/3138443500,v1:127.0.0.1:6813/3138443500] exists,up 7dae1f32-821a-4f5d-a555-819a510aee61 2024-06-03T05:26:28.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:26:28.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:26:28.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:26:28.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1686: test_wait_for_clean: wait_for_clean 2024-06-03T05:26:28.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T05:26:28.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T05:26:28.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T05:26:28.271 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T05:26:28.271 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:26:28.271 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:26:28.272 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:26:28.272 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:26:28.272 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:26:28.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T05:26:28.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T05:26:28.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T05:26:28.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T05:26:28.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T05:26:28.380 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T05:26:28.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T05:26:28.894 INFO:tasks.workunit.client.0.smithi088.stderr:1' 2024-06-03T05:26:28.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T05:26:28.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:26:28.894 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T05:26:29.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836492 2024-06-03T05:26:29.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836492 2024-06-03T05:26:29.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836492' 2024-06-03T05:26:29.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:26:29.053 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T05:26:29.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=68719476746 2024-06-03T05:26:29.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 68719476746 2024-06-03T05:26:29.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836492 1-68719476746' 2024-06-03T05:26:29.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:26:29.205 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836492 2024-06-03T05:26:29.206 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:26:29.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T05:26:29.207 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836492 2024-06-03T05:26:29.207 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:26:29.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836492 2024-06-03T05:26:29.208 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836492 2024-06-03T05:26:29.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-06-03T05:26:29.208 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:26:29.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836490 -lt 21474836492 2024-06-03T05:26:29.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T05:26:30.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T05:26:30.609 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:26:31.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836492 2024-06-03T05:26:31.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:26:31.136 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-68719476746 2024-06-03T05:26:31.137 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:26:31.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T05:26:31.138 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-68719476746 2024-06-03T05:26:31.139 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:26:31.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=68719476746 2024-06-03T05:26:31.140 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 68719476746 2024-06-03T05:26:31.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 68719476746' 2024-06-03T05:26:31.140 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T05:26:31.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 68719476746 -lt 68719476746 2024-06-03T05:26:31.550 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T05:26:31.550 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:26:31.550 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:26:32.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-03T05:26:32.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T05:26:32.250 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T05:26:32.250 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T05:26:32.250 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T05:26:32.250 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T05:26:32.250 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T05:26:32.250 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T05:26:32.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T05:26:32.635 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T05:26:32.635 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:26:32.635 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:26:33.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-03T05:26:33.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T05:26:33.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T05:26:33.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: test_wait_for_clean: teardown td/ceph-helpers 2024-06-03T05:26:33.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:26:33.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:26:33.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:26:33.161 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:26:33.162 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:26:33.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:26:33.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:26:33.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:26:33.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:26:33.271 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:26:33.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:26:33.273 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:26:33.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:26:33.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:26:33.274 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:26:33.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:26:33.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:26:33.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:26:33.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:26:33.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:26:33.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:26:33.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:26:33.286 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:26:33.286 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:26:33.286 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:26:33.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:26:33.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:26:33.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:26:33.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:26:33.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_wait_for_health_ok td/ceph-helpers 2024-06-03T05:26:33.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1806: test_wait_for_health_ok: local dir=td/ceph-helpers 2024-06-03T05:26:33.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1808: test_wait_for_health_ok: setup td/ceph-helpers 2024-06-03T05:26:33.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-03T05:26:33.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-03T05:26:33.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:26:33.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:26:33.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:26:33.292 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:26:33.292 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:26:33.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:26:33.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:26:33.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:26:33.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:26:33.293 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:26:33.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:26:33.293 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:26:33.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:26:33.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:26:33.294 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:26:33.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:26:33.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:26:33.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:26:33.294 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:26:33.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:26:33.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:26:33.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:26:33.296 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:26:33.296 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:26:33.296 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:26:33.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:26:33.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:26:33.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:26:33.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-03T05:26:33.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:26:33.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:26:33.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:26:33.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70897 2024-06-03T05:26:33.300 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:26:33.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:26:33.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:26:33.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-03T05:26:33.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1809: test_wait_for_health_ok: run_mon td/ceph-helpers a --osd_failsafe_full_ratio=.99 --mon_pg_warn_min_per_osd=0 2024-06-03T05:26:33.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-03T05:26:33.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:26:33.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:26:33.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:26:33.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-03T05:26:33.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_failsafe_full_ratio=.99 --mon_pg_warn_min_per_osd=0 2024-06-03T05:26:33.743 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:26:33.744 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:26:33.744 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:26:33.744 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:26:33.744 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:26:33.744 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:26:33.744 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:26:33.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_failsafe_full_ratio=.99 --mon_pg_warn_min_per_osd=0 2024-06-03T05:26:33.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:26:33.973 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:26:33.973 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:26:33.973 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:26:33.973 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:26:33.973 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:26:33.973 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:26:33.974 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:26:33.974 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:26:33.974 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:26:33.975 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:26:33.975 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:26:33.975 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:26:33.975 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:26:33.975 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get fsid 2024-06-03T05:26:34.090 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:26:34.090 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:26:34.090 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:26:34.090 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:26:34.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:26:34.092 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:26:34.092 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:26:34.092 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:26:34.092 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:26:34.092 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:26:34.093 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:26:34.093 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:26:34.093 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:26:34.093 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get mon_host 2024-06-03T05:26:34.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1810: test_wait_for_health_ok: run_mgr td/ceph-helpers x --mon_pg_warn_min_per_osd=0 2024-06-03T05:26:34.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-03T05:26:34.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T05:26:34.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T05:26:34.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T05:26:34.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-03T05:26:34.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T05:26:35.128 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T05:26:35.128 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:26:35.128 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:26:35.128 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:26:35.128 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:26:35.129 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:26:35.129 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:26:35.129 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:26:35.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mon_pg_warn_min_per_osd=0 2024-06-03T05:26:35.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1812: test_wait_for_health_ok: run_osd td/ceph-helpers 0 2024-06-03T05:26:35.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:26:35.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:26:35.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T05:26:35.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:26:35.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-03T05:26:35.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:26:35.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:26:35.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:26:35.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:26:35.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-03T05:26:35.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-03T05:26:35.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:26:35.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:26:35.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:26:35.167 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:26:35.167 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:26:35.167 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:26:35.167 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:26:35.167 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:26:35.168 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:26:35.168 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:26:35.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:26:35.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:26:35.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:26:35.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:26:35.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:26:35.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:26:35.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:26:35.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:26:35.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:26:35.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:26:35.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:26:35.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:26:35.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-03T05:26:35.169 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:26:35.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bcd56b19-b10f-4e52-a705-a8a96dba2c82 2024-06-03T05:26:35.170 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 bcd56b19-b10f-4e52-a705-a8a96dba2c82 2024-06-03T05:26:35.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 bcd56b19-b10f-4e52-a705-a8a96dba2c82' 2024-06-03T05:26:35.170 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:26:35.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQALVF1m9tbwChAAeD3XZJn01FleKX3Xy7XQFQ== 2024-06-03T05:26:35.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQALVF1m9tbwChAAeD3XZJn01FleKX3Xy7XQFQ=="}' 2024-06-03T05:26:35.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bcd56b19-b10f-4e52-a705-a8a96dba2c82 -i td/ceph-helpers/0/new.json 2024-06-03T05:26:35.459 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:26:35.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-03T05:26:35.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQALVF1m9tbwChAAeD3XZJn01FleKX3Xy7XQFQ== --osd-uuid bcd56b19-b10f-4e52-a705-a8a96dba2c82 2024-06-03T05:26:35.501 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:26:35.495+0000 7fb5b8e027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:26:35.519 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:26:35.515+0000 7fb5b8e027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:26:35.520 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:26:35.515+0000 7fb5b8e027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:26:35.520 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:26:35.515+0000 7fb5b8e027c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:26:35.521 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:26:35.515+0000 7fb5b8e027c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:26:35.521 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:26:35.515+0000 7fb5b8e027c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:26:35.551 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:26:35.547+0000 7fb5b8e027c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-03T05:26:41.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-03T05:26:41.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:26:41.893 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T05:26:41.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T05:26:41.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:26:42.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T05:26:42.436 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:26:42.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:26:42.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:26:42.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:26:42.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:26:42.468 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:26:42.463+0000 7f56220047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:26:42.470 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:26:42.467+0000 7f56220047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:26:42.471 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:26:42.467+0000 7f56220047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:26:42.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T05:26:42.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:26:42.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:26:42.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:26:42.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:26:42.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:26:42.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:26:42.859 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:26:42.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:26:42.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:26:43.047 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:26:43.043+0000 7f56220047c0 -1 Falling back to public interface 2024-06-03T05:26:43.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:26:44.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:26:44.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:26:44.283 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:26:44.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:26:44.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:26:44.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:26:44.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:26:45.636 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:26:45.631+0000 7f56220047c0 -1 osd.0 0 log_to_monitors true 2024-06-03T05:26:45.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:26:45.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:26:45.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:26:45.687 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:26:45.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:26:45.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:26:46.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:26:47.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:26:47.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:26:47.255 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:26:47.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:26:47.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:26:47.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:26:47.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:26:48.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:26:48.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:26:48.717 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:26:48.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:26:48.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:26:48.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:26:49.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:26:50.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:26:50.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:26:50.165 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:26:50.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:26:50.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:26:50.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:26:50.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:26:51.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:26:51.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:26:51.684 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:26:51.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:26:51.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:26:51.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:26:52.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:26:53.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:26:53.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:26:53.134 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:26:53.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:26:53.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:26:53.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:26:53.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:26:54.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:26:54.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:26:54.708 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:26:54.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:26:54.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:26:54.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:26:55.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:26:56.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:26:56.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:26:56.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:26:56.147 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:26:56.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:26:56.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:26:56.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:26:57.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:26:57.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:26:57.708 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:26:57.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:26:57.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:26:57.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:26:58.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:26:59.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:26:59.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:26:59.152 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:26:59.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:26:59.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:26:59.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:27:00.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:27:01.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:27:01.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:27:01.042 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:27:01.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:27:01.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:27:01.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:27:01.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:27:02.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:27:02.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:27:02.879 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:27:02.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:27:02.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:27:02.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:27:03.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:27:04.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:27:04.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:27:04.683 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:27:04.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:27:04.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:27:04.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:27:05.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:27:06.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:27:06.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:27:06.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:27:06.531 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:27:06.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:27:06.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:27:07.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:27:08.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:27:08.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:27:08.621 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:27:08.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:27:08.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:27:08.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:27:09.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:27:10.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:27:10.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:27:10.651 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:27:10.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:27:10.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:27:10.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:27:11.177 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3056172647,v1:127.0.0.1:6803/3056172647] [v2:127.0.0.1:6804/3056172647,v1:127.0.0.1:6805/3056172647] exists,up bcd56b19-b10f-4e52-a705-a8a96dba2c82 2024-06-03T05:27:11.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:27:11.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:27:11.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:27:11.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1813: test_wait_for_health_ok: run_osd td/ceph-helpers 1 2024-06-03T05:27:11.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:27:11.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:27:11.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-03T05:27:11.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:27:11.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-06-03T05:27:11.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:27:11.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:27:11.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:27:11.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:27:11.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-06-03T05:27:11.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-06-03T05:27:11.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:27:11.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:27:11.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:27:11.183 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:27:11.183 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:27:11.183 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:27:11.183 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:27:11.184 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:27:11.184 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:27:11.184 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:27:11.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:27:11.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:27:11.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:27:11.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:27:11.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:27:11.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:27:11.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:27:11.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:27:11.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:27:11.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:27:11.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:27:11.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:27:11.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-06-03T05:27:11.185 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:27:11.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0ce90dfe-48d8-47aa-a69d-51d7ce328d4a 2024-06-03T05:27:11.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 0ce90dfe-48d8-47aa-a69d-51d7ce328d4a' 2024-06-03T05:27:11.185 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 0ce90dfe-48d8-47aa-a69d-51d7ce328d4a 2024-06-03T05:27:11.186 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:27:11.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAvVF1mUmEODBAA9w3mC38Iqf7RM5FkN/vaxA== 2024-06-03T05:27:11.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAvVF1mUmEODBAA9w3mC38Iqf7RM5FkN/vaxA=="}' 2024-06-03T05:27:11.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0ce90dfe-48d8-47aa-a69d-51d7ce328d4a -i td/ceph-helpers/1/new.json 2024-06-03T05:27:11.676 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:27:11.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-06-03T05:27:11.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAvVF1mUmEODBAA9w3mC38Iqf7RM5FkN/vaxA== --osd-uuid 0ce90dfe-48d8-47aa-a69d-51d7ce328d4a 2024-06-03T05:27:11.725 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:27:11.723+0000 7f87851e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:27:11.796 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:27:11.791+0000 7f87851e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:27:11.798 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:27:11.795+0000 7f87851e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:27:11.799 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:27:11.795+0000 7f87851e07c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-03T05:27:11.799 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:27:11.795+0000 7f87851e07c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-03T05:27:11.799 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:27:11.795+0000 7f87851e07c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-03T05:27:11.904 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:27:11.899+0000 7f87851e07c0 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-06-03T05:27:18.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-06-03T05:27:18.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:27:18.400 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-06-03T05:27:18.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-03T05:27:18.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:27:19.082 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-06-03T05:27:19.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-03T05:27:19.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:27:19.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:27:19.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:27:19.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:27:19.115 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:27:19.111+0000 7fbfc5fcc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:27:19.145 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:27:19.139+0000 7fbfc5fcc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:27:19.146 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:27:19.143+0000 7fbfc5fcc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:27:19.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-03T05:27:19.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:27:19.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T05:27:19.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:27:19.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:27:19.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:27:19.508 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:27:19.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:27:19.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:27:19.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:27:19.707 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:27:19.703+0000 7fbfc5fcc7c0 -1 Falling back to public interface 2024-06-03T05:27:19.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:27:20.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:27:20.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:27:20.915 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:27:20.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:27:20.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:27:20.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:27:21.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:27:22.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:27:22.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:27:22.336 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:27:22.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:27:22.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:27:22.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:27:22.349 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:27:22.343+0000 7fbfc5fcc7c0 -1 osd.1 0 log_to_monitors true 2024-06-03T05:27:22.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:27:23.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:27:23.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:27:23.758 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:27:23.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:27:23.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:27:23.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:27:24.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:27:25.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:27:25.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:27:25.285 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:27:25.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:27:25.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:27:25.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:27:25.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:27:26.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:27:26.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:27:26.738 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:27:26.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:27:26.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:27:26.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:27:27.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:27:28.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:27:28.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:27:28.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:27:28.358 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:27:28.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:27:28.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:27:28.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:27:29.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:27:29.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:27:29.852 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:27:29.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:27:29.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:27:29.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:27:30.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:27:31.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:27:31.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:27:31.322 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:27:31.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:27:31.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:27:31.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:27:31.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:27:32.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:27:32.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:27:32.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:27:32.775 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:27:32.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:27:32.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:27:33.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:27:34.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:27:34.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:27:34.300 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:27:34.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:27:34.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:27:34.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:27:34.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:27:35.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:27:35.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:27:35.814 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:27:35.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:27:35.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:27:35.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:27:36.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:27:37.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:27:37.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:27:37.365 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:27:37.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:27:37.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:27:37.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:27:37.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:27:38.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:27:38.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:27:38.957 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:27:38.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:27:38.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:27:38.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:27:39.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:27:40.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:27:40.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:27:40.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:27:40.870 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:27:40.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:27:40.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:27:41.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:27:42.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:27:42.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:27:42.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:27:42.819 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:27:42.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:27:42.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:27:43.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:27:44.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:27:44.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:27:44.727 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:27:44.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:27:44.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:27:44.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:27:45.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:27:46.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:27:46.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:27:46.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:27:46.499 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:27:46.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:27:46.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:27:47.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:27:48.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:27:48.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:27:48.492 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-06-03T05:27:48.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T05:27:48.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:27:48.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:27:48.919 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/923916228,v1:127.0.0.1:6811/923916228] [v2:127.0.0.1:6812/923916228,v1:127.0.0.1:6813/923916228] exists,up 0ce90dfe-48d8-47aa-a69d-51d7ce328d4a 2024-06-03T05:27:48.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:27:48.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:27:48.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:27:48.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1814: test_wait_for_health_ok: run_osd td/ceph-helpers 2 2024-06-03T05:27:48.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:27:48.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:27:48.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-03T05:27:48.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:27:48.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/2 2024-06-03T05:27:48.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:27:48.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:27:48.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:27:48.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:27:48.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2024-06-03T05:27:48.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2024-06-03T05:27:48.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:27:48.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:27:48.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:27:48.922 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:27:48.922 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:27:48.922 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:27:48.922 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:27:48.922 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:27:48.922 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:27:48.923 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:27:48.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:27:48.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:27:48.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:27:48.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:27:48.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:27:48.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:27:48.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:27:48.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:27:48.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:27:48.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:27:48.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:27:48.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:27:48.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/2 2024-06-03T05:27:48.925 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:27:48.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a1964d09-2178-4f7b-b41d-7f04c34a1462 2024-06-03T05:27:48.925 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 a1964d09-2178-4f7b-b41d-7f04c34a1462 2024-06-03T05:27:48.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 a1964d09-2178-4f7b-b41d-7f04c34a1462' 2024-06-03T05:27:48.925 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:27:48.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBUVF1m3d9HOBAAIfdpDvHuozhBwqi1kk/Jnw== 2024-06-03T05:27:48.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBUVF1m3d9HOBAAIfdpDvHuozhBwqi1kk/Jnw=="}' 2024-06-03T05:27:48.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a1964d09-2178-4f7b-b41d-7f04c34a1462 -i td/ceph-helpers/2/new.json 2024-06-03T05:27:49.629 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:27:49.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/2/new.json 2024-06-03T05:27:49.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBUVF1m3d9HOBAAIfdpDvHuozhBwqi1kk/Jnw== --osd-uuid a1964d09-2178-4f7b-b41d-7f04c34a1462 2024-06-03T05:27:49.681 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:27:49.675+0000 7fb62872d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:27:49.788 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:27:49.783+0000 7fb62872d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:27:49.790 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:27:49.787+0000 7fb62872d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:27:49.790 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:27:49.787+0000 7fb62872d7c0 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-06-03T05:27:49.790 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:27:49.787+0000 7fb62872d7c0 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-06-03T05:27:49.790 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:27:49.787+0000 7fb62872d7c0 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-06-03T05:27:49.866 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:27:49.863+0000 7fb62872d7c0 -1 bluestore(td/ceph-helpers/2) _read_fsid unparsable uuid 2024-06-03T05:27:56.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/2/keyring 2024-06-03T05:27:56.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:27:56.585 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-06-03T05:27:56.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-03T05:27:56.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:27:57.123 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-06-03T05:27:57.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-03T05:27:57.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:27:57.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:27:57.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:27:57.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:27:57.154 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:27:57.151+0000 7ff7c998c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:27:57.179 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:27:57.175+0000 7ff7c998c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:27:57.180 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:27:57.175+0000 7ff7c998c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:27:57.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-03T05:27:57.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:27:57.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T05:27:57.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:27:57.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:27:57.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:27:57.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:27:57.535 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:27:57.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:27:57.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:27:57.732 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:27:57.727+0000 7ff7c998c7c0 -1 Falling back to public interface 2024-06-03T05:27:57.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:27:58.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:27:58.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:27:58.949 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:27:58.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:27:58.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:27:58.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:27:59.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:28:00.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:28:00.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:28:00.370 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:28:00.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:28:00.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:28:00.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:28:00.412 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:28:00.407+0000 7ff7c998c7c0 -1 osd.2 0 log_to_monitors true 2024-06-03T05:28:00.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:28:01.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:28:01.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:28:01.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:28:01.789 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:28:01.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:28:01.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:28:02.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:28:03.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:28:03.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:28:03.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:28:03.303 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:28:03.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:28:03.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:28:03.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:28:04.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:28:04.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:28:04.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:28:04.764 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:28:04.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:28:04.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:28:05.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:28:06.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:28:06.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:28:06.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:28:06.325 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:28:06.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:28:06.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:28:06.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:28:07.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:28:07.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:28:07.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:28:07.766 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:28:07.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:28:07.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:28:08.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:28:09.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:28:09.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:28:09.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:28:09.311 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:28:09.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:28:09.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:28:09.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:28:10.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:28:10.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:28:10.773 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:28:10.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:28:10.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:28:10.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:28:11.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:28:12.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:28:12.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:28:12.347 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:28:12.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:28:12.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:28:12.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:28:12.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:28:13.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:28:13.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:28:13.777 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:28:13.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:28:13.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:28:13.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:28:14.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:28:15.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:28:15.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:28:15.276 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:28:15.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:28:15.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:28:15.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:28:15.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:28:16.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:28:16.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:28:16.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:28:16.867 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:28:16.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:28:16.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:28:17.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:28:18.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:28:18.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:28:18.695 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:28:18.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:28:18.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:28:18.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:28:19.513 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:28:20.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:28:20.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:28:20.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:28:20.515 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:28:20.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:28:20.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:28:21.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:28:22.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:28:22.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:28:22.640 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:28:22.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:28:22.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:28:22.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:28:23.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:28:24.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:28:24.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:28:24.529 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:28:24.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:28:24.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:28:24.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:28:25.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:28:26.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:28:26.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:28:26.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T05:28:26.326 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-06-03T05:28:26.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:28:26.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:28:26.794 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/645980211,v1:127.0.0.1:6819/645980211] [v2:127.0.0.1:6820/645980211,v1:127.0.0.1:6821/645980211] exists,up a1964d09-2178-4f7b-b41d-7f04c34a1462 2024-06-03T05:28:26.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:28:26.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:28:26.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:28:26.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1815: test_wait_for_health_ok: kill_daemons td/ceph-helpers TERM osd 2024-06-03T05:28:26.795 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:28:26.795 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:28:26.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:28:26.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:28:26.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:28:27.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:28:27.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1816: test_wait_for_health_ok: ceph osd down 0 2024-06-03T05:28:27.519 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already down. 2024-06-03T05:28:27.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1818: test_wait_for_health_ok: TIMEOUT=1 2024-06-03T05:28:27.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1818: test_wait_for_health_ok: wait_for_health_ok 2024-06-03T05:28:27.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1802: wait_for_health_ok: wait_for_health HEALTH_OK 2024-06-03T05:28:27.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1781: wait_for_health: local grepstr=HEALTH_OK 2024-06-03T05:28:27.536 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1782: wait_for_health: get_timeout_delays 1 .1 2024-06-03T05:28:27.537 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:28:27.537 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:28:27.537 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:28:27.537 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:28:27.537 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:28:27.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1782: wait_for_health: delays=('0.1' '0.2' '0.4' '0.3') 2024-06-03T05:28:27.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1782: wait_for_health: local -a delays 2024-06-03T05:28:27.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1783: wait_for_health: local -i loop=0 2024-06-03T05:28:27.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1785: wait_for_health: ceph health detail 2024-06-03T05:28:27.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1785: wait_for_health: grep HEALTH_OK 2024-06-03T05:28:28.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1786: wait_for_health: (( 0 >= 4 )) 2024-06-03T05:28:28.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_health: sleep 0.1 2024-06-03T05:28:28.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_health: loop+=1 2024-06-03T05:28:28.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1785: wait_for_health: ceph health detail 2024-06-03T05:28:28.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1785: wait_for_health: grep HEALTH_OK 2024-06-03T05:28:28.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1786: wait_for_health: (( 1 >= 4 )) 2024-06-03T05:28:28.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_health: sleep 0.2 2024-06-03T05:28:28.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_health: loop+=1 2024-06-03T05:28:28.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1785: wait_for_health: ceph health detail 2024-06-03T05:28:28.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1785: wait_for_health: grep HEALTH_OK 2024-06-03T05:28:29.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1786: wait_for_health: (( 2 >= 4 )) 2024-06-03T05:28:29.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_health: sleep 0.4 2024-06-03T05:28:29.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_health: loop+=1 2024-06-03T05:28:29.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1785: wait_for_health: ceph health detail 2024-06-03T05:28:29.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1785: wait_for_health: grep HEALTH_OK 2024-06-03T05:28:30.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1786: wait_for_health: (( 3 >= 4 )) 2024-06-03T05:28:30.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_health: sleep 0.3 2024-06-03T05:28:30.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_health: loop+=1 2024-06-03T05:28:30.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1785: wait_for_health: ceph health detail 2024-06-03T05:28:30.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1785: wait_for_health: grep HEALTH_OK 2024-06-03T05:28:31.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1786: wait_for_health: (( 4 >= 4 )) 2024-06-03T05:28:31.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1787: wait_for_health: ceph health detail 2024-06-03T05:28:31.865 INFO:tasks.workunit.client.0.smithi088.stdout:HEALTH_WARN 3 osds down; 1 host (3 osds) down; 1 root (3 osds) down 2024-06-03T05:28:31.866 INFO:tasks.workunit.client.0.smithi088.stdout:[WRN] OSD_DOWN: 3 osds down 2024-06-03T05:28:31.866 INFO:tasks.workunit.client.0.smithi088.stdout: osd.0 (root=default,host=smithi088) is down 2024-06-03T05:28:31.866 INFO:tasks.workunit.client.0.smithi088.stdout: osd.1 (root=default,host=smithi088) is down 2024-06-03T05:28:31.866 INFO:tasks.workunit.client.0.smithi088.stdout: osd.2 (root=default,host=smithi088) is down 2024-06-03T05:28:31.866 INFO:tasks.workunit.client.0.smithi088.stdout:[WRN] OSD_HOST_DOWN: 1 host (3 osds) down 2024-06-03T05:28:31.866 INFO:tasks.workunit.client.0.smithi088.stdout: host smithi088 (root=default) (3 osds) is down 2024-06-03T05:28:31.866 INFO:tasks.workunit.client.0.smithi088.stdout:[WRN] OSD_ROOT_DOWN: 1 root (3 osds) down 2024-06-03T05:28:31.866 INFO:tasks.workunit.client.0.smithi088.stdout: root default (3 osds) is down 2024-06-03T05:28:31.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1788: wait_for_health: return 1 2024-06-03T05:28:31.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1802: wait_for_health_ok: return 1 2024-06-03T05:28:31.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1820: test_wait_for_health_ok: activate_osd td/ceph-helpers 0 2024-06-03T05:28:31.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-06-03T05:28:31.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T05:28:31.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T05:28:31.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T05:28:31.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-06-03T05:28:31.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:28:31.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:28:31.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:28:31.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:28:31.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-03T05:28:31.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-03T05:28:31.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T05:28:31.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T05:28:31.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:28:31.887 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T05:28:31.887 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:28:31.887 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:28:31.888 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:28:31.888 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:28:31.888 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:28:31.888 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:28:31.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:28:31.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:28:31.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:28:31.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:28:31.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:28:31.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:28:31.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:28:31.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:28:31.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T05:28:31.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T05:28:31.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-06-03T05:28:31.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T05:28:31.890 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:28:31.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:28:31.891 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-06-03T05:28:31.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T05:28:31.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T05:28:31.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T05:28:31.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T05:28:31.920 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:28:31.915+0000 7f1e217797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:28:31.947 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:28:31.943+0000 7f1e217797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:28:31.948 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:28:31.943+0000 7f1e217797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:28:32.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T05:28:32.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:28:32.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:28:32.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:28:32.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:28:32.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:28:32.300 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:28:32.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:28:32.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:28:32.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:28:32.503 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:28:32.499+0000 7f1e217797c0 -1 Falling back to public interface 2024-06-03T05:28:32.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:28:33.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:28:33.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:28:33.716 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:28:33.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:28:33.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:28:33.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:28:34.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:28:35.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:28:35.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:28:35.126 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:28:35.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:28:35.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:28:35.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:28:35.371 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:28:35.367+0000 7f1e217797c0 -1 osd.0 14 log_to_monitors true 2024-06-03T05:28:35.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:28:36.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:28:36.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:28:36.613 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:28:36.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:28:36.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:28:36.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:28:37.024 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 19 up_thru 0 down_at 17 last_clean_interval [5,14) [v2:127.0.0.1:6802/1764957480,v1:127.0.0.1:6803/1764957480] [v2:127.0.0.1:6804/1764957480,v1:127.0.0.1:6805/1764957480] exists,up bcd56b19-b10f-4e52-a705-a8a96dba2c82 2024-06-03T05:28:37.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:28:37.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:28:37.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:28:37.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1821: test_wait_for_health_ok: activate_osd td/ceph-helpers 1 2024-06-03T05:28:37.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-06-03T05:28:37.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T05:28:37.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T05:28:37.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T05:28:37.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/1 2024-06-03T05:28:37.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:28:37.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:28:37.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:28:37.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:28:37.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-06-03T05:28:37.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-06-03T05:28:37.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T05:28:37.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T05:28:37.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:28:37.029 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T05:28:37.029 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:28:37.030 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:28:37.030 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:28:37.030 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:28:37.030 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:28:37.030 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:28:37.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:28:37.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:28:37.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:28:37.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:28:37.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:28:37.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:28:37.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:28:37.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:28:37.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T05:28:37.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T05:28:37.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/1 2024-06-03T05:28:37.031 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-06-03T05:28:37.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T05:28:37.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:28:37.033 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/1/whoami 2024-06-03T05:28:37.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T05:28:37.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T05:28:37.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T05:28:37.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T05:28:37.061 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:28:37.055+0000 7fe2d686f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:28:37.069 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:28:37.063+0000 7fe2d686f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:28:37.070 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:28:37.067+0000 7fe2d686f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:28:37.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T05:28:37.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:28:37.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T05:28:37.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:28:37.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:28:37.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:28:37.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:28:37.473 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:28:37.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:28:37.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:28:37.639 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:28:37.635+0000 7fe2d686f7c0 -1 Falling back to public interface 2024-06-03T05:28:37.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:28:38.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:28:38.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:28:38.929 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:28:38.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:28:38.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:28:38.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:28:39.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:28:40.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:28:40.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:28:40.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:28:40.339 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:28:40.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:28:40.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:28:40.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:28:40.794 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:28:40.791+0000 7fe2d686f7c0 -1 osd.1 16 log_to_monitors true 2024-06-03T05:28:41.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:28:41.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:28:41.755 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:28:41.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:28:41.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:28:41.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:28:42.165 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 21 up_thru 0 down_at 17 last_clean_interval [10,16) [v2:127.0.0.1:6810/3931061623,v1:127.0.0.1:6811/3931061623] [v2:127.0.0.1:6812/3931061623,v1:127.0.0.1:6813/3931061623] exists,up 0ce90dfe-48d8-47aa-a69d-51d7ce328d4a 2024-06-03T05:28:42.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:28:42.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:28:42.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:28:42.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1822: test_wait_for_health_ok: activate_osd td/ceph-helpers 2 2024-06-03T05:28:42.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-06-03T05:28:42.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T05:28:42.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-06-03T05:28:42.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T05:28:42.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/2 2024-06-03T05:28:42.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:28:42.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:28:42.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:28:42.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:28:42.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2024-06-03T05:28:42.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2024-06-03T05:28:42.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T05:28:42.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T05:28:42.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:28:42.173 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T05:28:42.173 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:28:42.173 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:28:42.173 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:28:42.173 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:28:42.173 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:28:42.173 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:28:42.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:28:42.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:28:42.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:28:42.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:28:42.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:28:42.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:28:42.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:28:42.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:28:42.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T05:28:42.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T05:28:42.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/2 2024-06-03T05:28:42.175 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-06-03T05:28:42.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-06-03T05:28:42.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:28:42.176 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/2/whoami 2024-06-03T05:28:42.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-06-03T05:28:42.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T05:28:42.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T05:28:42.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T05:28:42.213 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:28:42.207+0000 7fe8c766e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:28:42.215 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:28:42.211+0000 7fe8c766e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:28:42.216 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:28:42.211+0000 7fe8c766e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:28:42.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-06-03T05:28:42.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:28:42.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T05:28:42.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:28:42.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:28:42.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:28:42.607 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:28:42.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:28:42.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:28:42.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:28:42.807 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:28:42.803+0000 7fe8c766e7c0 -1 Falling back to public interface 2024-06-03T05:28:43.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:28:44.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:28:44.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:28:44.028 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:28:44.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:28:44.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:28:44.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:28:44.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:28:45.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:28:45.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:28:45.471 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:28:45.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:28:45.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:28:45.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:28:45.909 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:28:45.903+0000 7fe8c766e7c0 -1 osd.2 15 log_to_monitors true 2024-06-03T05:28:45.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:28:46.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:28:46.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:28:46.943 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:28:46.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:28:46.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:28:46.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:28:47.364 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 23 up_thru 0 down_at 17 last_clean_interval [0,0) [v2:127.0.0.1:6818/1490537156,v1:127.0.0.1:6819/1490537156] [v2:127.0.0.1:6820/1490537156,v1:127.0.0.1:6821/1490537156] exists,up a1964d09-2178-4f7b-b41d-7f04c34a1462 2024-06-03T05:28:47.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:28:47.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:28:47.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:28:47.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1823: test_wait_for_health_ok: wait_for_health_ok 2024-06-03T05:28:47.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1802: wait_for_health_ok: wait_for_health HEALTH_OK 2024-06-03T05:28:47.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1781: wait_for_health: local grepstr=HEALTH_OK 2024-06-03T05:28:47.366 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1782: wait_for_health: get_timeout_delays 300 .1 2024-06-03T05:28:47.366 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:28:47.366 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:28:47.367 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:28:47.367 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:28:47.367 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:28:47.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1782: wait_for_health: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '4.5') 2024-06-03T05:28:47.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1782: wait_for_health: local -a delays 2024-06-03T05:28:47.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1783: wait_for_health: local -i loop=0 2024-06-03T05:28:47.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1785: wait_for_health: ceph health detail 2024-06-03T05:28:47.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1785: wait_for_health: grep HEALTH_OK 2024-06-03T05:28:48.171 INFO:tasks.workunit.client.0.smithi088.stdout:HEALTH_OK 2024-06-03T05:28:48.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1824: test_wait_for_health_ok: teardown td/ceph-helpers 2024-06-03T05:28:48.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:28:48.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:28:48.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:28:48.172 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:28:48.172 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:28:48.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:28:48.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:28:48.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:28:48.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:28:48.281 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:28:48.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:28:48.281 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:28:48.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:28:48.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:28:48.283 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:28:48.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:28:48.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:28:48.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:28:48.284 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:28:48.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:28:48.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:28:48.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:28:48.300 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:28:48.300 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:28:48.300 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:28:48.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:28:48.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:28:48.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:28:48.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:28:48.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_wait_for_osd td/ceph-helpers 2024-06-03T05:28:48.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:995: test_wait_for_osd: local dir=td/ceph-helpers 2024-06-03T05:28:48.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:996: test_wait_for_osd: setup td/ceph-helpers 2024-06-03T05:28:48.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-03T05:28:48.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-03T05:28:48.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:28:48.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:28:48.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:28:48.303 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:28:48.303 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:28:48.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:28:48.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:28:48.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:28:48.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:28:48.304 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:28:48.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:28:48.307 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:28:48.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:28:48.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:28:48.309 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:28:48.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:28:48.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:28:48.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:28:48.310 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:28:48.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:28:48.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:28:48.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:28:48.313 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:28:48.313 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:28:48.313 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:28:48.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:28:48.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:28:48.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:28:48.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-03T05:28:48.317 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:28:48.317 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:28:48.317 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:28:48.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70897 2024-06-03T05:28:48.318 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:28:48.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:28:48.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:28:48.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-03T05:28:48.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:997: test_wait_for_osd: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:28:48.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-03T05:28:48.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:28:48.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:28:48.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:28:48.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-03T05:28:48.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:28:48.734 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:28:48.734 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:28:48.734 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:28:48.735 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:28:48.735 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:28:48.735 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:28:48.735 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:28:48.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:28:48.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:28:48.944 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:28:48.944 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:28:48.944 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:28:48.945 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:28:48.945 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:28:48.945 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:28:48.945 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:28:48.945 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:28:48.946 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:28:48.946 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:28:48.946 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:28:48.946 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:28:48.947 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:28:48.947 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get fsid 2024-06-03T05:28:49.055 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:28:49.055 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:28:49.055 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:28:49.056 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:28:49.057 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:28:49.057 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:28:49.057 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:28:49.057 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:28:49.057 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:28:49.057 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:28:49.057 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:28:49.057 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:28:49.058 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:28:49.058 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get mon_host 2024-06-03T05:28:49.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:998: test_wait_for_osd: run_mgr td/ceph-helpers x 2024-06-03T05:28:49.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-03T05:28:49.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T05:28:49.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T05:28:49.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T05:28:49.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-03T05:28:49.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T05:28:50.091 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T05:28:50.091 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:28:50.091 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:28:50.091 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:28:50.092 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:28:50.092 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:28:50.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:28:50.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:28:50.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:28:50.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:999: test_wait_for_osd: run_osd td/ceph-helpers 0 2024-06-03T05:28:50.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:28:50.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:28:50.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T05:28:50.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:28:50.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-03T05:28:50.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:28:50.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:28:50.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:28:50.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:28:50.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-03T05:28:50.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-03T05:28:50.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:28:50.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:28:50.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:28:50.129 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:28:50.129 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:28:50.129 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:28:50.132 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:28:50.132 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:28:50.132 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:28:50.132 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:28:50.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:28:50.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:28:50.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:28:50.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:28:50.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:28:50.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:28:50.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:28:50.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:28:50.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:28:50.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:28:50.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:28:50.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:28:50.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-03T05:28:50.134 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:28:50.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f732273d-b8ee-4247-b361-c02c38ecf497 2024-06-03T05:28:50.134 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 f732273d-b8ee-4247-b361-c02c38ecf497 2024-06-03T05:28:50.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f732273d-b8ee-4247-b361-c02c38ecf497' 2024-06-03T05:28:50.135 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:28:50.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCSVF1mIgndCBAAFIFGH3/jFyFMTqheWTuQ2Q== 2024-06-03T05:28:50.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCSVF1mIgndCBAAFIFGH3/jFyFMTqheWTuQ2Q=="}' 2024-06-03T05:28:50.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f732273d-b8ee-4247-b361-c02c38ecf497 -i td/ceph-helpers/0/new.json 2024-06-03T05:28:50.475 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:28:50.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-03T05:28:50.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCSVF1mIgndCBAAFIFGH3/jFyFMTqheWTuQ2Q== --osd-uuid f732273d-b8ee-4247-b361-c02c38ecf497 2024-06-03T05:28:50.531 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:28:50.527+0000 7f86e1c6a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:28:50.551 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:28:50.547+0000 7f86e1c6a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:28:50.553 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:28:50.547+0000 7f86e1c6a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:28:50.554 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:28:50.551+0000 7f86e1c6a7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:28:50.554 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:28:50.551+0000 7f86e1c6a7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:28:50.554 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:28:50.551+0000 7f86e1c6a7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:28:50.600 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:28:50.595+0000 7f86e1c6a7c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-03T05:28:56.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-03T05:28:56.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:28:56.939 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T05:28:56.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T05:28:56.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:28:57.491 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:28:57.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T05:28:57.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:28:57.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:28:57.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:28:57.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:28:57.523 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:28:57.519+0000 7f763e1437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:28:57.525 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:28:57.519+0000 7f763e1437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:28:57.527 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:28:57.523+0000 7f763e1437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:28:57.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T05:28:57.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:28:57.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:28:57.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:28:57.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:28:57.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:28:57.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:28:57.927 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:28:57.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:28:57.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:28:58.083 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:28:58.079+0000 7f763e1437c0 -1 Falling back to public interface 2024-06-03T05:28:58.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:28:59.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:28:59.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:28:59.376 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:28:59.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:28:59.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:28:59.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:28:59.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:29:00.632 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:29:00.627+0000 7f763e1437c0 -1 osd.0 0 log_to_monitors true 2024-06-03T05:29:00.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:29:00.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:29:00.796 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:29:00.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:29:00.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:29:00.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:29:01.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:29:02.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:29:02.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:29:02.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:29:02.426 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:29:02.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:29:02.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:29:02.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:29:03.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:29:03.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:29:03.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:29:03.852 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:29:03.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:29:03.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:29:04.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:29:05.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:29:05.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:29:05.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:29:05.279 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:29:05.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:29:05.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:29:05.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:29:06.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:29:06.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:29:06.861 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:29:06.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:29:06.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:29:06.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:29:07.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:29:08.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:29:08.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:29:08.298 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:29:08.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:29:08.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:29:08.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:29:08.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:29:09.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:29:09.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:29:09.876 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:29:09.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:29:09.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:29:09.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:29:10.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:29:11.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:29:11.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:29:11.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:29:11.297 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:29:11.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:29:11.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:29:11.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:29:12.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:29:12.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:29:12.873 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:29:12.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:29:12.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:29:12.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:29:13.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:29:14.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:29:14.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:29:14.297 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:29:14.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:29:14.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:29:14.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:29:15.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:29:16.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:29:16.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:29:16.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:29:16.151 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:29:16.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:29:16.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:29:17.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:29:18.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:29:18.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:29:18.077 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:29:18.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:29:18.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:29:18.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:29:18.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:29:19.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:29:19.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:29:19.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:29:19.848 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:29:19.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:29:19.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:29:20.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:29:21.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:29:21.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:29:21.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:29:21.783 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:29:21.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:29:21.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:29:22.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:29:23.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:29:23.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:29:23.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:29:23.804 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:29:23.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:29:23.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:29:24.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:29:25.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:29:25.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:29:25.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:29:25.657 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:29:25.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:29:25.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:29:26.085 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3962729541,v1:127.0.0.1:6803/3962729541] [v2:127.0.0.1:6804/3962729541,v1:127.0.0.1:6805/3962729541] exists,up f732273d-b8ee-4247-b361-c02c38ecf497 2024-06-03T05:29:26.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:29:26.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:29:26.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:29:26.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1000: test_wait_for_osd: run_osd td/ceph-helpers 1 2024-06-03T05:29:26.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:29:26.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:29:26.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-03T05:29:26.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:29:26.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-06-03T05:29:26.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:29:26.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:29:26.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:29:26.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:29:26.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-06-03T05:29:26.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-06-03T05:29:26.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:29:26.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:29:26.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:29:26.089 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:29:26.089 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:29:26.089 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:29:26.089 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:29:26.089 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:29:26.089 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:29:26.089 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:29:26.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:29:26.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:29:26.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:29:26.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:29:26.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:29:26.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:29:26.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:29:26.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:29:26.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:29:26.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:29:26.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:29:26.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:29:26.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-06-03T05:29:26.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:29:26.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1b29c88b-d567-47d6-b1cd-6f53a133afea 2024-06-03T05:29:26.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 1b29c88b-d567-47d6-b1cd-6f53a133afea' 2024-06-03T05:29:26.092 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 1b29c88b-d567-47d6-b1cd-6f53a133afea 2024-06-03T05:29:26.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:29:26.113 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC2VF1mpfOYBhAA6UTsK5o8DTHQDu4GHO2QFQ== 2024-06-03T05:29:26.113 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC2VF1mpfOYBhAA6UTsK5o8DTHQDu4GHO2QFQ=="}' 2024-06-03T05:29:26.113 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1b29c88b-d567-47d6-b1cd-6f53a133afea -i td/ceph-helpers/1/new.json 2024-06-03T05:29:26.735 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:29:26.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-06-03T05:29:26.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC2VF1mpfOYBhAA6UTsK5o8DTHQDu4GHO2QFQ== --osd-uuid 1b29c88b-d567-47d6-b1cd-6f53a133afea 2024-06-03T05:29:26.791 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:29:26.787+0000 7f38a91e67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:29:26.799 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:29:26.795+0000 7f38a91e67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:29:26.800 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:29:26.795+0000 7f38a91e67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:29:26.801 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:29:26.795+0000 7f38a91e67c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-03T05:29:26.801 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:29:26.795+0000 7f38a91e67c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-03T05:29:26.801 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:29:26.795+0000 7f38a91e67c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-03T05:29:26.877 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:29:26.871+0000 7f38a91e67c0 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-06-03T05:29:33.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-06-03T05:29:33.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:29:33.620 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-06-03T05:29:33.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-03T05:29:33.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:29:34.163 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-06-03T05:29:34.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-03T05:29:34.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:29:34.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:29:34.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:29:34.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:29:34.195 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:29:34.190+0000 7fbf1c9417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:29:34.214 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:29:34.210+0000 7fbf1c9417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:29:34.215 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:29:34.210+0000 7fbf1c9417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:29:34.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-03T05:29:34.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:29:34.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T05:29:34.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:29:34.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:29:34.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:29:34.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:29:34.627 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:29:34.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:29:34.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:29:34.795 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:29:34.790+0000 7fbf1c9417c0 -1 Falling back to public interface 2024-06-03T05:29:35.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:29:36.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:29:36.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:29:36.043 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:29:36.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:29:36.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:29:36.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:29:36.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:29:37.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:29:37.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:29:37.458 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:29:37.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:29:37.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:29:37.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:29:37.470 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:29:37.466+0000 7fbf1c9417c0 -1 osd.1 0 log_to_monitors true 2024-06-03T05:29:38.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:29:39.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:29:39.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:29:39.215 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:29:39.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:29:39.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:29:39.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:29:39.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:29:40.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:29:40.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:29:40.701 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:29:40.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:29:40.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:29:40.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:29:41.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:29:42.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:29:42.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:29:42.167 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:29:42.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:29:42.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:29:42.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:29:42.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:29:43.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:29:43.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:29:43.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:29:43.751 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:29:43.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:29:43.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:29:44.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:29:45.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:29:45.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:29:45.175 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:29:45.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:29:45.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:29:45.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:29:45.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:29:46.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:29:46.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:29:46.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:29:46.655 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:29:46.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:29:46.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:29:47.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:29:48.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:29:48.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:29:48.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:29:48.154 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:29:48.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:29:48.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:29:48.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:29:49.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:29:49.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:29:49.650 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:29:49.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:29:49.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:29:49.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:29:50.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:29:51.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:29:51.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:29:51.098 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:29:51.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:29:51.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:29:51.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:29:52.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:29:53.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:29:53.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:29:53.191 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:29:53.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:29:53.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:29:53.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:29:54.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:29:55.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:29:55.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:29:55.386 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:29:55.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:29:55.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:29:55.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:29:56.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:29:57.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:29:57.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:29:57.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:29:57.169 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:29:57.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:29:57.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:29:58.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:29:59.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:29:59.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:29:59.332 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:29:59.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:29:59.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:29:59.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:30:00.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:30:01.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:30:01.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:30:01.139 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:30:01.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:30:01.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:30:01.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:30:02.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:30:03.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:30:03.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:30:03.013 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:30:03.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:30:03.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:30:03.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:30:03.414 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2804259712,v1:127.0.0.1:6811/2804259712] [v2:127.0.0.1:6812/2804259712,v1:127.0.0.1:6813/2804259712] exists,up 1b29c88b-d567-47d6-b1cd-6f53a133afea 2024-06-03T05:30:03.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:30:03.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:30:03.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:30:03.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1001: test_wait_for_osd: wait_for_osd up 0 2024-06-03T05:30:03.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:30:03.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:30:03.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:30:03.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:30:03.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:30:03.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:30:03.416 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:30:03.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:30:03.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:30:03.871 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3962729541,v1:127.0.0.1:6803/3962729541] [v2:127.0.0.1:6804/3962729541,v1:127.0.0.1:6805/3962729541] exists,up f732273d-b8ee-4247-b361-c02c38ecf497 2024-06-03T05:30:03.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:30:03.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:30:03.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:30:03.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1002: test_wait_for_osd: wait_for_osd up 1 2024-06-03T05:30:03.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:30:03.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T05:30:03.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:30:03.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:30:03.873 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:30:03.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:30:03.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:30:03.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:30:03.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:30:04.304 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2804259712,v1:127.0.0.1:6811/2804259712] [v2:127.0.0.1:6812/2804259712,v1:127.0.0.1:6813/2804259712] exists,up 1b29c88b-d567-47d6-b1cd-6f53a133afea 2024-06-03T05:30:04.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:30:04.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:30:04.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:30:04.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1003: test_wait_for_osd: kill_daemons td/ceph-helpers TERM osd.0 2024-06-03T05:30:04.305 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:30:04.305 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:30:04.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:30:04.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:30:04.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:30:04.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:30:04.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1004: test_wait_for_osd: wait_for_osd down 0 2024-06-03T05:30:04.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2024-06-03T05:30:04.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:30:04.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:30:04.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:30:04.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:30:04.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:30:04.616 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:30:04.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:30:04.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2024-06-03T05:30:05.036 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 12 last_clean_interval [0,0) [v2:127.0.0.1:6802/3962729541,v1:127.0.0.1:6803/3962729541] [v2:127.0.0.1:6804/3962729541,v1:127.0.0.1:6805/3962729541] exists f732273d-b8ee-4247-b361-c02c38ecf497 2024-06-03T05:30:05.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:30:05.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:30:05.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:30:05.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1005: test_wait_for_osd: TIMEOUT=1 2024-06-03T05:30:05.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1005: test_wait_for_osd: wait_for_osd up 0 2024-06-03T05:30:05.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:30:05.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:30:05.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:30:05.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:30:05.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 1 )) 2024-06-03T05:30:05.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:30:05.038 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:30:05.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:30:05.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:30:05.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:30:06.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:30:06.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 1 )) 2024-06-03T05:30:06.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 1 2024-06-03T05:30:06.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1006: test_wait_for_osd: teardown td/ceph-helpers 2024-06-03T05:30:06.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:30:06.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:30:06.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:30:06.478 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:30:06.478 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:30:06.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:30:06.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:30:06.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:30:06.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:30:06.791 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:30:06.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:30:06.792 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:30:06.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:30:06.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:30:06.794 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:30:06.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:30:06.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:30:06.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:30:06.797 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:30:06.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:30:06.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:30:06.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:30:06.812 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:30:06.812 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:30:06.812 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:30:06.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:30:06.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:30:06.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:30:06.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:30:06.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_wait_for_peered td/ceph-helpers 2024-06-03T05:30:06.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1734: test_wait_for_peered: local dir=td/ceph-helpers 2024-06-03T05:30:06.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1736: test_wait_for_peered: setup td/ceph-helpers 2024-06-03T05:30:06.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-03T05:30:06.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-03T05:30:06.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:30:06.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:30:06.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:30:06.816 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:30:06.816 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:30:06.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:30:06.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:30:06.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:30:06.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:30:06.817 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:30:06.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:30:06.818 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:30:06.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:30:06.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:30:06.820 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:30:06.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:30:06.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:30:06.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:30:06.822 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:30:06.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:30:06.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:30:06.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:30:06.825 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:30:06.825 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:30:06.826 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:30:06.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:30:06.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:30:06.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:30:06.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-03T05:30:06.828 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:30:06.828 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:30:06.829 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:30:06.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70897 2024-06-03T05:30:06.831 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:30:06.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:30:06.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:30:06.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-03T05:30:06.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1737: test_wait_for_peered: run_mon td/ceph-helpers a --osd_pool_default_size=2 2024-06-03T05:30:06.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-03T05:30:06.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:30:06.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:30:06.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:30:06.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-03T05:30:06.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2024-06-03T05:30:07.250 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:30:07.250 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:30:07.250 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:30:07.250 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:30:07.250 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:30:07.250 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:30:07.251 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:30:07.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-06-03T05:30:07.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:30:07.472 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:30:07.472 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:30:07.472 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:30:07.472 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:30:07.473 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:30:07.473 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:30:07.473 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:30:07.473 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:30:07.474 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:30:07.474 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:30:07.474 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:30:07.474 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:30:07.474 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:30:07.475 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get fsid 2024-06-03T05:30:07.592 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:30:07.592 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:30:07.592 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:30:07.592 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:30:07.592 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:30:07.593 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:30:07.593 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:30:07.594 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:30:07.594 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:30:07.594 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:30:07.594 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:30:07.594 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:30:07.594 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:30:07.595 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get mon_host 2024-06-03T05:30:07.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1738: test_wait_for_peered: run_osd td/ceph-helpers 0 2024-06-03T05:30:07.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:30:07.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:30:07.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T05:30:07.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:30:07.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-03T05:30:07.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:30:07.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:30:07.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:30:07.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:30:07.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-03T05:30:07.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-03T05:30:07.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:30:07.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:30:07.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:30:07.710 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:30:07.710 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:30:07.710 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:30:07.710 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:30:07.710 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:30:07.710 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:30:07.710 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:30:07.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:30:07.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:30:07.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:30:07.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:30:07.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:30:07.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:30:07.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:30:07.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:30:07.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:30:07.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:30:07.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:30:07.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:30:07.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-03T05:30:07.711 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:30:07.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=294ae4c0-1f85-4aef-bad7-02ff27b34dad 2024-06-03T05:30:07.712 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 294ae4c0-1f85-4aef-bad7-02ff27b34dad 2024-06-03T05:30:07.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 294ae4c0-1f85-4aef-bad7-02ff27b34dad' 2024-06-03T05:30:07.712 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:30:07.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDfVF1mlL1lKxAAIvJ09npSgZnbVehcSVL5Fg== 2024-06-03T05:30:07.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDfVF1mlL1lKxAAIvJ09npSgZnbVehcSVL5Fg=="}' 2024-06-03T05:30:07.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 294ae4c0-1f85-4aef-bad7-02ff27b34dad -i td/ceph-helpers/0/new.json 2024-06-03T05:30:08.694 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:30:08.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-03T05:30:08.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDfVF1mlL1lKxAAIvJ09npSgZnbVehcSVL5Fg== --osd-uuid 294ae4c0-1f85-4aef-bad7-02ff27b34dad 2024-06-03T05:30:08.743 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:30:08.738+0000 7fc97216a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:30:08.745 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:30:08.738+0000 7fc97216a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:30:08.746 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:30:08.742+0000 7fc97216a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:30:08.747 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:30:08.742+0000 7fc97216a7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:30:08.747 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:30:08.742+0000 7fc97216a7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:30:08.747 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:30:08.742+0000 7fc97216a7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:30:08.840 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:30:08.834+0000 7fc97216a7c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-03T05:30:14.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-03T05:30:14.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:30:14.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T05:30:14.981 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T05:30:14.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:30:15.227 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:30:15.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T05:30:15.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:30:15.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:30:15.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:30:15.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:30:15.260 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:30:15.254+0000 7f9d2af037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:30:15.262 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:30:15.258+0000 7f9d2af037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:30:15.264 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:30:15.258+0000 7f9d2af037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:30:15.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T05:30:15.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:30:15.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:30:15.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:30:15.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:30:15.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:30:15.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:30:15.458 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:30:15.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:30:15.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:30:15.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:30:15.825 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:30:15.818+0000 7f9d2af037c0 -1 Falling back to public interface 2024-06-03T05:30:16.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:30:16.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:30:16.702 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:30:16.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:30:16.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:30:16.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:30:16.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:30:17.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:30:17.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:30:17.947 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:30:17.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:30:17.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:30:17.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:30:18.116 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:30:18.110+0000 7f9d2af037c0 -1 osd.0 0 log_to_monitors true 2024-06-03T05:30:18.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:30:19.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:30:19.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:30:19.231 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:30:19.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:30:19.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:30:19.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:30:19.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:30:20.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:30:20.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:30:20.465 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:30:20.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:30:20.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:30:20.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:30:20.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:30:21.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:30:21.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:30:21.698 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:30:21.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:30:21.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:30:21.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:30:21.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:30:22.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:30:22.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:30:22.927 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:30:22.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:30:22.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:30:22.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:30:23.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:30:24.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:30:24.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:30:24.165 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:30:24.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:30:24.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:30:24.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:30:24.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:30:25.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:30:25.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:30:25.403 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:30:25.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:30:25.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:30:25.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:30:25.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:30:26.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:30:26.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:30:26.642 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:30:26.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:30:26.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:30:26.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:30:26.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:30:27.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:30:27.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:30:27.883 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:30:27.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:30:27.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:30:27.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:30:28.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:30:29.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:30:29.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:30:29.117 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:30:29.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:30:29.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:30:29.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:30:29.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:30:30.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:30:30.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:30:30.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:30:30.345 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:30:30.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:30:30.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:30:30.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:30:31.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:30:31.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:30:31.582 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:30:31.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:30:31.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:30:31.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:30:31.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:30:32.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:30:32.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:30:32.821 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:30:32.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:30:32.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:30:32.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:30:33.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:30:34.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:30:34.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:30:34.057 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:30:34.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:30:34.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:30:34.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:30:34.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:30:35.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:30:35.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:30:35.287 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:30:35.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:30:35.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:30:35.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:30:35.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:30:36.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:30:36.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:30:36.530 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:30:36.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:30:36.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:30:36.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:30:36.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:30:37.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:30:37.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:30:37.792 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-06-03T05:30:37.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T05:30:37.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:30:37.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:30:38.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:30:39.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:30:39.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:30:39.007 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-06-03T05:30:39.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-03T05:30:39.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:30:39.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:30:39.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:30:40.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:30:40.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:30:40.241 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-06-03T05:30:40.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-03T05:30:40.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:30:40.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:30:40.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:30:41.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:30:41.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:30:41.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-03T05:30:41.490 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-06-03T05:30:41.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:30:41.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:30:41.726 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1160640607,v1:127.0.0.1:6801/1160640607] [v2:127.0.0.1:6802/1160640607,v1:127.0.0.1:6803/1160640607] exists,up 294ae4c0-1f85-4aef-bad7-02ff27b34dad 2024-06-03T05:30:41.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:30:41.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:30:41.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:30:41.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1739: test_wait_for_peered: run_mgr td/ceph-helpers x 2024-06-03T05:30:41.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-03T05:30:41.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T05:30:41.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T05:30:41.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T05:30:41.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-03T05:30:41.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T05:30:42.023 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T05:30:42.024 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:30:42.024 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:30:42.025 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:30:42.025 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:30:42.025 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:30:42.025 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:30:42.025 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:30:42.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:30:42.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1740: test_wait_for_peered: create_rbd_pool 2024-06-03T05:30:42.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-03T05:30:42.287 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' does not exist 2024-06-03T05:30:42.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-03T05:30:42.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-03T05:30:42.709 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' already exists 2024-06-03T05:30:42.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T05:30:43.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-03T05:30:44.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1741: test_wait_for_peered: WAIT_FOR_CLEAN_TIMEOUT=1 2024-06-03T05:30:44.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1741: test_wait_for_peered: wait_for_clean 2024-06-03T05:30:44.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T05:30:44.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T05:30:44.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T05:30:44.819 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 1 .1 2024-06-03T05:30:44.819 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:30:44.819 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:30:44.819 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:30:44.819 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:30:44.819 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:30:44.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.3') 2024-06-03T05:30:44.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T05:30:44.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T05:30:44.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T05:30:44.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T05:30:44.862 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T05:30:45.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-06-03T05:30:45.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T05:30:45.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:30:45.097 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T05:30:45.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=0 2024-06-03T05:30:45.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 0 2024-06-03T05:30:45.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-0' 2024-06-03T05:30:45.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:30:45.245 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-0 2024-06-03T05:30:45.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:30:45.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T05:30:45.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-0 2024-06-03T05:30:45.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:30:45.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=0 2024-06-03T05:30:45.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 0' 2024-06-03T05:30:45.248 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 0 2024-06-03T05:30:45.248 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:30:45.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 0 -lt 0 2024-06-03T05:30:45.468 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T05:30:45.468 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:30:45.468 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:30:45.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 == 0 2024-06-03T05:30:45.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: sleep 1 2024-06-03T05:30:46.699 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T05:30:46.699 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:30:46.699 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:30:47.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-03T05:30:47.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T05:30:47.327 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T05:30:47.327 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T05:30:47.327 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T05:30:47.327 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T05:30:47.327 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T05:30:47.328 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T05:30:47.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-03T05:30:47.740 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T05:30:47.740 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:30:47.740 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:30:48.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-06-03T05:30:48.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-06-03T05:30:48.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-03T05:30:48.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-06-03T05:30:48.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T05:30:48.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-03T05:30:48.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T05:30:48.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T05:30:48.355 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T05:30:48.355 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T05:30:48.356 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T05:30:48.356 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T05:30:48.356 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T05:30:48.356 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T05:30:48.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-03T05:30:48.748 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T05:30:48.748 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:30:48.748 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:30:49.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-06-03T05:30:49.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-06-03T05:30:49.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-03T05:30:49.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T05:30:49.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T05:30:49.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T05:30:49.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T05:30:49.259 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T05:30:49.259 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T05:30:49.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-03T05:30:49.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-03T05:30:49.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 4 )) 2024-06-03T05:30:49.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T05:30:49.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-03T05:30:49.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T05:30:49.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T05:30:49.960 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T05:30:49.960 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T05:30:49.960 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T05:30:49.960 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T05:30:49.960 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T05:30:49.961 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T05:30:50.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-03T05:30:50.359 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T05:30:50.359 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:30:50.359 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:30:50.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-06-03T05:30:50.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-06-03T05:30:50.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-03T05:30:50.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T05:30:50.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T05:30:50.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T05:30:50.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T05:30:50.931 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T05:30:50.931 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T05:30:51.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-03T05:30:51.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-03T05:30:51.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 4 )) 2024-06-03T05:30:51.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T05:30:51.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-06-03T05:30:51.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T05:30:51.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T05:30:51.858 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T05:30:51.859 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T05:30:51.859 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T05:30:51.859 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T05:30:51.859 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T05:30:51.859 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T05:30:52.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-03T05:30:52.260 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T05:30:52.261 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:30:52.261 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:30:52.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-06-03T05:30:52.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-06-03T05:30:52.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-03T05:30:52.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T05:30:52.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T05:30:52.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T05:30:52.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T05:30:52.808 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T05:30:52.808 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T05:30:53.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-03T05:30:53.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-03T05:30:53.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 4 )) 2024-06-03T05:30:53.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-03T05:30:53.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.3 2024-06-03T05:30:53.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-03T05:30:53.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T05:30:53.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T05:30:53.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T05:30:53.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T05:30:53.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T05:30:53.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T05:30:53.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T05:30:54.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-03T05:30:54.030 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T05:30:54.030 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:30:54.030 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:30:54.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-06-03T05:30:54.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-06-03T05:30:54.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-03T05:30:54.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-03T05:30:54.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-03T05:30:54.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-03T05:30:54.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-03T05:30:54.553 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-03T05:30:54.553 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-03T05:30:55.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-03T05:30:55.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-03T05:30:55.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 4 )) 2024-06-03T05:30:55.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-06-03T05:30:55.623 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-06-03T05:30:55.624 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_fingerprint": "0283d120-098e-42bf-8837-16b42f6ecc8d", 2024-06-03T05:30:55.624 INFO:tasks.workunit.client.0.smithi088.stdout: "version": "19.0.0-2946-g3d89accf", 2024-06-03T05:30:55.624 INFO:tasks.workunit.client.0.smithi088.stdout: "commit": "3d89accfe3201286f000e065a1faea2a88c7a6ba", 2024-06-03T05:30:55.624 INFO:tasks.workunit.client.0.smithi088.stdout: "timestamp": "2024-06-03T05:30:55.620097+0000", 2024-06-03T05:30:55.624 INFO:tasks.workunit.client.0.smithi088.stdout: "tag": "", 2024-06-03T05:30:55.624 INFO:tasks.workunit.client.0.smithi088.stdout: "health": { 2024-06-03T05:30:55.624 INFO:tasks.workunit.client.0.smithi088.stdout: "status": "HEALTH_WARN", 2024-06-03T05:30:55.624 INFO:tasks.workunit.client.0.smithi088.stdout: "checks": { 2024-06-03T05:30:55.624 INFO:tasks.workunit.client.0.smithi088.stdout: "PG_DEGRADED": { 2024-06-03T05:30:55.624 INFO:tasks.workunit.client.0.smithi088.stdout: "severity": "HEALTH_WARN", 2024-06-03T05:30:55.624 INFO:tasks.workunit.client.0.smithi088.stdout: "summary": { 2024-06-03T05:30:55.624 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "Degraded data redundancy: 1/2 objects degraded (50.000%), 1 pg degraded", 2024-06-03T05:30:55.625 INFO:tasks.workunit.client.0.smithi088.stdout: "count": 1 2024-06-03T05:30:55.625 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.625 INFO:tasks.workunit.client.0.smithi088.stdout: "detail": [ 2024-06-03T05:30:55.625 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.625 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0]" 2024-06-03T05:30:55.625 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:30:55.625 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-06-03T05:30:55.625 INFO:tasks.workunit.client.0.smithi088.stdout: "muted": false 2024-06-03T05:30:55.625 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.625 INFO:tasks.workunit.client.0.smithi088.stdout: "TOO_FEW_OSDS": { 2024-06-03T05:30:55.625 INFO:tasks.workunit.client.0.smithi088.stdout: "severity": "HEALTH_WARN", 2024-06-03T05:30:55.625 INFO:tasks.workunit.client.0.smithi088.stdout: "summary": { 2024-06-03T05:30:55.625 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "OSD count 1 < osd_pool_default_size 3", 2024-06-03T05:30:55.626 INFO:tasks.workunit.client.0.smithi088.stdout: "count": 2 2024-06-03T05:30:55.626 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.626 INFO:tasks.workunit.client.0.smithi088.stdout: "detail": [], 2024-06-03T05:30:55.626 INFO:tasks.workunit.client.0.smithi088.stdout: "muted": false 2024-06-03T05:30:55.626 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:30:55.626 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.626 INFO:tasks.workunit.client.0.smithi088.stdout: "mutes": [] 2024-06-03T05:30:55.626 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.626 INFO:tasks.workunit.client.0.smithi088.stdout: "monmap_first_committed": 1, 2024-06-03T05:30:55.626 INFO:tasks.workunit.client.0.smithi088.stdout: "monmap_last_committed": 1, 2024-06-03T05:30:55.626 INFO:tasks.workunit.client.0.smithi088.stdout: "monmap": { 2024-06-03T05:30:55.626 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 1, 2024-06-03T05:30:55.626 INFO:tasks.workunit.client.0.smithi088.stdout: "fsid": "213fb4bd-379a-40f7-a383-f6c59451eb9c", 2024-06-03T05:30:55.626 INFO:tasks.workunit.client.0.smithi088.stdout: "modified": "2024-06-03T05:30:06.857576Z", 2024-06-03T05:30:55.627 INFO:tasks.workunit.client.0.smithi088.stdout: "created": "2024-06-03T05:30:06.857576Z", 2024-06-03T05:30:55.627 INFO:tasks.workunit.client.0.smithi088.stdout: "min_mon_release": 19, 2024-06-03T05:30:55.627 INFO:tasks.workunit.client.0.smithi088.stdout: "min_mon_release_name": "squid", 2024-06-03T05:30:55.627 INFO:tasks.workunit.client.0.smithi088.stdout: "election_strategy": 1, 2024-06-03T05:30:55.627 INFO:tasks.workunit.client.0.smithi088.stdout: "disallowed_leaders: ": "", 2024-06-03T05:30:55.627 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_mode": false, 2024-06-03T05:30:55.627 INFO:tasks.workunit.client.0.smithi088.stdout: "tiebreaker_mon": "", 2024-06-03T05:30:55.627 INFO:tasks.workunit.client.0.smithi088.stdout: "removed_ranks: ": "", 2024-06-03T05:30:55.627 INFO:tasks.workunit.client.0.smithi088.stdout: "features": { 2024-06-03T05:30:55.627 INFO:tasks.workunit.client.0.smithi088.stdout: "persistent": [ 2024-06-03T05:30:55.627 INFO:tasks.workunit.client.0.smithi088.stdout: "kraken", 2024-06-03T05:30:55.627 INFO:tasks.workunit.client.0.smithi088.stdout: "luminous", 2024-06-03T05:30:55.627 INFO:tasks.workunit.client.0.smithi088.stdout: "mimic", 2024-06-03T05:30:55.627 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap-prune", 2024-06-03T05:30:55.628 INFO:tasks.workunit.client.0.smithi088.stdout: "nautilus", 2024-06-03T05:30:55.628 INFO:tasks.workunit.client.0.smithi088.stdout: "octopus", 2024-06-03T05:30:55.628 INFO:tasks.workunit.client.0.smithi088.stdout: "pacific", 2024-06-03T05:30:55.628 INFO:tasks.workunit.client.0.smithi088.stdout: "elector-pinging", 2024-06-03T05:30:55.628 INFO:tasks.workunit.client.0.smithi088.stdout: "quincy", 2024-06-03T05:30:55.628 INFO:tasks.workunit.client.0.smithi088.stdout: "reef", 2024-06-03T05:30:55.628 INFO:tasks.workunit.client.0.smithi088.stdout: "squid" 2024-06-03T05:30:55.628 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-06-03T05:30:55.628 INFO:tasks.workunit.client.0.smithi088.stdout: "optional": [] 2024-06-03T05:30:55.628 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.628 INFO:tasks.workunit.client.0.smithi088.stdout: "mons": [ 2024-06-03T05:30:55.628 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.628 INFO:tasks.workunit.client.0.smithi088.stdout: "rank": 0, 2024-06-03T05:30:55.628 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "a", 2024-06-03T05:30:55.629 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addrs": { 2024-06-03T05:30:55.629 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-06-03T05:30:55.629 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.629 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-06-03T05:30:55.629 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:7109", 2024-06-03T05:30:55.629 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 0 2024-06-03T05:30:55.629 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:30:55.629 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-06-03T05:30:55.629 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.629 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:7109/0", 2024-06-03T05:30:55.629 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addr": "127.0.0.1:7109/0", 2024-06-03T05:30:55.629 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 0, 2024-06-03T05:30:55.629 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 0, 2024-06-03T05:30:55.629 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_location": "{}" 2024-06-03T05:30:55.629 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:30:55.630 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-06-03T05:30:55.630 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.630 INFO:tasks.workunit.client.0.smithi088.stdout: "quorum": [ 2024-06-03T05:30:55.630 INFO:tasks.workunit.client.0.smithi088.stdout: 0 2024-06-03T05:30:55.630 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-06-03T05:30:55.630 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap": { 2024-06-03T05:30:55.630 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 12, 2024-06-03T05:30:55.630 INFO:tasks.workunit.client.0.smithi088.stdout: "fsid": "213fb4bd-379a-40f7-a383-f6c59451eb9c", 2024-06-03T05:30:55.630 INFO:tasks.workunit.client.0.smithi088.stdout: "created": "2024-06-03T05:30:07.818366+0000", 2024-06-03T05:30:55.630 INFO:tasks.workunit.client.0.smithi088.stdout: "modified": "2024-06-03T05:30:44.484166+0000", 2024-06-03T05:30:55.630 INFO:tasks.workunit.client.0.smithi088.stdout: "last_up_change": "2024-06-03T05:30:40.828225+0000", 2024-06-03T05:30:55.630 INFO:tasks.workunit.client.0.smithi088.stdout: "last_in_change": "2024-06-03T05:30:08.525259+0000", 2024-06-03T05:30:55.630 INFO:tasks.workunit.client.0.smithi088.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-06-03T05:30:55.630 INFO:tasks.workunit.client.0.smithi088.stdout: "flags_num": 5799936, 2024-06-03T05:30:55.631 INFO:tasks.workunit.client.0.smithi088.stdout: "flags_set": [ 2024-06-03T05:30:55.631 INFO:tasks.workunit.client.0.smithi088.stdout: "pglog_hardlimit", 2024-06-03T05:30:55.631 INFO:tasks.workunit.client.0.smithi088.stdout: "purged_snapdirs", 2024-06-03T05:30:55.631 INFO:tasks.workunit.client.0.smithi088.stdout: "recovery_deletes", 2024-06-03T05:30:55.631 INFO:tasks.workunit.client.0.smithi088.stdout: "sortbitwise" 2024-06-03T05:30:55.631 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-06-03T05:30:55.631 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_version": 3, 2024-06-03T05:30:55.631 INFO:tasks.workunit.client.0.smithi088.stdout: "full_ratio": 0.99000000953674316, 2024-06-03T05:30:55.631 INFO:tasks.workunit.client.0.smithi088.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-06-03T05:30:55.631 INFO:tasks.workunit.client.0.smithi088.stdout: "nearfull_ratio": 0.85000002384185791, 2024-06-03T05:30:55.631 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_snapshot": "", 2024-06-03T05:30:55.631 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_max": 1, 2024-06-03T05:30:55.631 INFO:tasks.workunit.client.0.smithi088.stdout: "max_osd": 1, 2024-06-03T05:30:55.631 INFO:tasks.workunit.client.0.smithi088.stdout: "require_min_compat_client": "luminous", 2024-06-03T05:30:55.632 INFO:tasks.workunit.client.0.smithi088.stdout: "min_compat_client": "jewel", 2024-06-03T05:30:55.632 INFO:tasks.workunit.client.0.smithi088.stdout: "require_osd_release": "squid", 2024-06-03T05:30:55.632 INFO:tasks.workunit.client.0.smithi088.stdout: "allow_crimson": false, 2024-06-03T05:30:55.632 INFO:tasks.workunit.client.0.smithi088.stdout: "pools": [ 2024-06-03T05:30:55.632 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.632 INFO:tasks.workunit.client.0.smithi088.stdout: "pool": 1, 2024-06-03T05:30:55.632 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_name": "rbd", 2024-06-03T05:30:55.632 INFO:tasks.workunit.client.0.smithi088.stdout: "create_time": "2024-06-03T05:30:42.493256+0000", 2024-06-03T05:30:55.632 INFO:tasks.workunit.client.0.smithi088.stdout: "flags": 8193, 2024-06-03T05:30:55.632 INFO:tasks.workunit.client.0.smithi088.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-06-03T05:30:55.632 INFO:tasks.workunit.client.0.smithi088.stdout: "type": 1, 2024-06-03T05:30:55.633 INFO:tasks.workunit.client.0.smithi088.stdout: "size": 2, 2024-06-03T05:30:55.633 INFO:tasks.workunit.client.0.smithi088.stdout: "min_size": 1, 2024-06-03T05:30:55.633 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_rule": 0, 2024-06-03T05:30:55.633 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_count": 0, 2024-06-03T05:30:55.633 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_target": 0, 2024-06-03T05:30:55.633 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_barrier": 0, 2024-06-03T05:30:55.633 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-06-03T05:30:55.633 INFO:tasks.workunit.client.0.smithi088.stdout: "object_hash": 2, 2024-06-03T05:30:55.633 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_autoscale_mode": "off", 2024-06-03T05:30:55.633 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_num": 4, 2024-06-03T05:30:55.633 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_placement_num": 4, 2024-06-03T05:30:55.633 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_placement_num_target": 4, 2024-06-03T05:30:55.633 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_num_target": 4, 2024-06-03T05:30:55.634 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_num_pending": 4, 2024-06-03T05:30:55.634 INFO:tasks.workunit.client.0.smithi088.stdout: "last_pg_merge_meta": { 2024-06-03T05:30:55.634 INFO:tasks.workunit.client.0.smithi088.stdout: "source_pgid": "0.0", 2024-06-03T05:30:55.634 INFO:tasks.workunit.client.0.smithi088.stdout: "ready_epoch": 0, 2024-06-03T05:30:55.634 INFO:tasks.workunit.client.0.smithi088.stdout: "last_epoch_started": 0, 2024-06-03T05:30:55.634 INFO:tasks.workunit.client.0.smithi088.stdout: "last_epoch_clean": 0, 2024-06-03T05:30:55.634 INFO:tasks.workunit.client.0.smithi088.stdout: "source_version": "0'0", 2024-06-03T05:30:55.634 INFO:tasks.workunit.client.0.smithi088.stdout: "target_version": "0'0" 2024-06-03T05:30:55.634 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.634 INFO:tasks.workunit.client.0.smithi088.stdout: "last_change": "12", 2024-06-03T05:30:55.634 INFO:tasks.workunit.client.0.smithi088.stdout: "last_force_op_resend": "0", 2024-06-03T05:30:55.634 INFO:tasks.workunit.client.0.smithi088.stdout: "last_force_op_resend_prenautilus": "0", 2024-06-03T05:30:55.634 INFO:tasks.workunit.client.0.smithi088.stdout: "last_force_op_resend_preluminous": "0", 2024-06-03T05:30:55.635 INFO:tasks.workunit.client.0.smithi088.stdout: "auid": 0, 2024-06-03T05:30:55.635 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_mode": "selfmanaged", 2024-06-03T05:30:55.635 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_seq": 2, 2024-06-03T05:30:55.635 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_epoch": 12, 2024-06-03T05:30:55.635 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_snaps": [], 2024-06-03T05:30:55.635 INFO:tasks.workunit.client.0.smithi088.stdout: "removed_snaps": "[]", 2024-06-03T05:30:55.635 INFO:tasks.workunit.client.0.smithi088.stdout: "quota_max_bytes": 0, 2024-06-03T05:30:55.635 INFO:tasks.workunit.client.0.smithi088.stdout: "quota_max_objects": 0, 2024-06-03T05:30:55.635 INFO:tasks.workunit.client.0.smithi088.stdout: "tiers": [], 2024-06-03T05:30:55.635 INFO:tasks.workunit.client.0.smithi088.stdout: "tier_of": -1, 2024-06-03T05:30:55.635 INFO:tasks.workunit.client.0.smithi088.stdout: "read_tier": -1, 2024-06-03T05:30:55.635 INFO:tasks.workunit.client.0.smithi088.stdout: "write_tier": -1, 2024-06-03T05:30:55.635 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_mode": "none", 2024-06-03T05:30:55.635 INFO:tasks.workunit.client.0.smithi088.stdout: "target_max_bytes": 0, 2024-06-03T05:30:55.636 INFO:tasks.workunit.client.0.smithi088.stdout: "target_max_objects": 0, 2024-06-03T05:30:55.636 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-06-03T05:30:55.636 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-06-03T05:30:55.636 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_target_full_ratio_micro": 800000, 2024-06-03T05:30:55.636 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_min_flush_age": 0, 2024-06-03T05:30:55.636 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_min_evict_age": 0, 2024-06-03T05:30:55.636 INFO:tasks.workunit.client.0.smithi088.stdout: "erasure_code_profile": "", 2024-06-03T05:30:55.636 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_params": { 2024-06-03T05:30:55.636 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "none" 2024-06-03T05:30:55.636 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.636 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_period": 0, 2024-06-03T05:30:55.636 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_count": 0, 2024-06-03T05:30:55.636 INFO:tasks.workunit.client.0.smithi088.stdout: "use_gmt_hitset": true, 2024-06-03T05:30:55.637 INFO:tasks.workunit.client.0.smithi088.stdout: "min_read_recency_for_promote": 0, 2024-06-03T05:30:55.637 INFO:tasks.workunit.client.0.smithi088.stdout: "min_write_recency_for_promote": 0, 2024-06-03T05:30:55.637 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_grade_decay_rate": 0, 2024-06-03T05:30:55.637 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_search_last_n": 0, 2024-06-03T05:30:55.637 INFO:tasks.workunit.client.0.smithi088.stdout: "grade_table": [], 2024-06-03T05:30:55.637 INFO:tasks.workunit.client.0.smithi088.stdout: "stripe_width": 0, 2024-06-03T05:30:55.637 INFO:tasks.workunit.client.0.smithi088.stdout: "expected_num_objects": 0, 2024-06-03T05:30:55.637 INFO:tasks.workunit.client.0.smithi088.stdout: "fast_read": false, 2024-06-03T05:30:55.637 INFO:tasks.workunit.client.0.smithi088.stdout: "options": {}, 2024-06-03T05:30:55.637 INFO:tasks.workunit.client.0.smithi088.stdout: "application_metadata": { 2024-06-03T05:30:55.637 INFO:tasks.workunit.client.0.smithi088.stdout: "rbd": {} 2024-06-03T05:30:55.638 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.638 INFO:tasks.workunit.client.0.smithi088.stdout: "read_balance": { 2024-06-03T05:30:55.638 INFO:tasks.workunit.client.0.smithi088.stdout: "score_type": "Fair distribution", 2024-06-03T05:30:55.638 INFO:tasks.workunit.client.0.smithi088.stdout: "score_acting": 1, 2024-06-03T05:30:55.638 INFO:tasks.workunit.client.0.smithi088.stdout: "score_stable": 1, 2024-06-03T05:30:55.638 INFO:tasks.workunit.client.0.smithi088.stdout: "optimal_score": 1, 2024-06-03T05:30:55.638 INFO:tasks.workunit.client.0.smithi088.stdout: "raw_score_acting": 1, 2024-06-03T05:30:55.638 INFO:tasks.workunit.client.0.smithi088.stdout: "raw_score_stable": 1, 2024-06-03T05:30:55.638 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_affinity_weighted": 1, 2024-06-03T05:30:55.638 INFO:tasks.workunit.client.0.smithi088.stdout: "average_primary_affinity": 1, 2024-06-03T05:30:55.638 INFO:tasks.workunit.client.0.smithi088.stdout: "average_primary_affinity_weighted": 1 2024-06-03T05:30:55.638 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:30:55.638 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:30:55.638 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-06-03T05:30:55.639 INFO:tasks.workunit.client.0.smithi088.stdout: "osds": [ 2024-06-03T05:30:55.639 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.639 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 0, 2024-06-03T05:30:55.639 INFO:tasks.workunit.client.0.smithi088.stdout: "uuid": "294ae4c0-1f85-4aef-bad7-02ff27b34dad", 2024-06-03T05:30:55.639 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 1, 2024-06-03T05:30:55.639 INFO:tasks.workunit.client.0.smithi088.stdout: "in": 1, 2024-06-03T05:30:55.639 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 1, 2024-06-03T05:30:55.639 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_affinity": 1, 2024-06-03T05:30:55.639 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_begin": 0, 2024-06-03T05:30:55.639 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_end": 0, 2024-06-03T05:30:55.639 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 5, 2024-06-03T05:30:55.639 INFO:tasks.workunit.client.0.smithi088.stdout: "up_thru": 7, 2024-06-03T05:30:55.639 INFO:tasks.workunit.client.0.smithi088.stdout: "down_at": 0, 2024-06-03T05:30:55.639 INFO:tasks.workunit.client.0.smithi088.stdout: "lost_at": 0, 2024-06-03T05:30:55.640 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addrs": { 2024-06-03T05:30:55.640 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-06-03T05:30:55.640 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.640 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-06-03T05:30:55.640 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6800", 2024-06-03T05:30:55.640 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1160640607 2024-06-03T05:30:55.640 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.640 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.640 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-06-03T05:30:55.640 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6801", 2024-06-03T05:30:55.640 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1160640607 2024-06-03T05:30:55.640 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:30:55.640 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-06-03T05:30:55.640 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.640 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addrs": { 2024-06-03T05:30:55.641 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-06-03T05:30:55.641 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.641 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-06-03T05:30:55.641 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6802", 2024-06-03T05:30:55.641 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1160640607 2024-06-03T05:30:55.641 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.641 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.641 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-06-03T05:30:55.641 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6803", 2024-06-03T05:30:55.641 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1160640607 2024-06-03T05:30:55.641 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:30:55.641 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-06-03T05:30:55.641 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.641 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addrs": { 2024-06-03T05:30:55.642 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-06-03T05:30:55.642 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.642 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-06-03T05:30:55.642 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6806", 2024-06-03T05:30:55.642 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1160640607 2024-06-03T05:30:55.642 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.642 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.642 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-06-03T05:30:55.642 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6807", 2024-06-03T05:30:55.642 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1160640607 2024-06-03T05:30:55.642 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:30:55.642 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-06-03T05:30:55.642 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.643 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addrs": { 2024-06-03T05:30:55.643 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-06-03T05:30:55.643 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.643 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-06-03T05:30:55.643 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6804", 2024-06-03T05:30:55.643 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1160640607 2024-06-03T05:30:55.643 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.643 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.643 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-06-03T05:30:55.643 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6805", 2024-06-03T05:30:55.643 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1160640607 2024-06-03T05:30:55.643 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:30:55.643 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-06-03T05:30:55.644 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.644 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addr": "127.0.0.1:6801/1160640607", 2024-06-03T05:30:55.644 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addr": "127.0.0.1:6803/1160640607", 2024-06-03T05:30:55.644 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addr": "127.0.0.1:6807/1160640607", 2024-06-03T05:30:55.644 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addr": "127.0.0.1:6805/1160640607", 2024-06-03T05:30:55.644 INFO:tasks.workunit.client.0.smithi088.stdout: "state": [ 2024-06-03T05:30:55.644 INFO:tasks.workunit.client.0.smithi088.stdout: "exists", 2024-06-03T05:30:55.644 INFO:tasks.workunit.client.0.smithi088.stdout: "up" 2024-06-03T05:30:55.644 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-06-03T05:30:55.644 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:30:55.644 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-06-03T05:30:55.644 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_xinfo": [ 2024-06-03T05:30:55.644 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.644 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 0, 2024-06-03T05:30:55.644 INFO:tasks.workunit.client.0.smithi088.stdout: "down_stamp": "0.000000", 2024-06-03T05:30:55.645 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_probability": 0, 2024-06-03T05:30:55.645 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_interval": 0, 2024-06-03T05:30:55.645 INFO:tasks.workunit.client.0.smithi088.stdout: "features": 4540701547738038271, 2024-06-03T05:30:55.645 INFO:tasks.workunit.client.0.smithi088.stdout: "old_weight": 0, 2024-06-03T05:30:55.645 INFO:tasks.workunit.client.0.smithi088.stdout: "last_purged_snaps_scrub": "2024-06-03T05:30:40.391557+0000", 2024-06-03T05:30:55.645 INFO:tasks.workunit.client.0.smithi088.stdout: "dead_epoch": 0 2024-06-03T05:30:55.645 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:30:55.645 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-06-03T05:30:55.645 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_upmap": [], 2024-06-03T05:30:55.645 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_upmap_items": [], 2024-06-03T05:30:55.645 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_upmap_primaries": [], 2024-06-03T05:30:55.645 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_temp": [], 2024-06-03T05:30:55.645 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_temp": [], 2024-06-03T05:30:55.645 INFO:tasks.workunit.client.0.smithi088.stdout: "blocklist": {}, 2024-06-03T05:30:55.645 INFO:tasks.workunit.client.0.smithi088.stdout: "range_blocklist": {}, 2024-06-03T05:30:55.646 INFO:tasks.workunit.client.0.smithi088.stdout: "erasure_code_profiles": { 2024-06-03T05:30:55.646 INFO:tasks.workunit.client.0.smithi088.stdout: "default": { 2024-06-03T05:30:55.646 INFO:tasks.workunit.client.0.smithi088.stdout: "k": "2", 2024-06-03T05:30:55.646 INFO:tasks.workunit.client.0.smithi088.stdout: "m": "2", 2024-06-03T05:30:55.646 INFO:tasks.workunit.client.0.smithi088.stdout: "plugin": "jerasure", 2024-06-03T05:30:55.646 INFO:tasks.workunit.client.0.smithi088.stdout: "technique": "reed_sol_van" 2024-06-03T05:30:55.646 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:30:55.646 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.646 INFO:tasks.workunit.client.0.smithi088.stdout: "removed_snaps_queue": [ 2024-06-03T05:30:55.646 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.646 INFO:tasks.workunit.client.0.smithi088.stdout: "pool": 1, 2024-06-03T05:30:55.646 INFO:tasks.workunit.client.0.smithi088.stdout: "snaps": [ 2024-06-03T05:30:55.646 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.646 INFO:tasks.workunit.client.0.smithi088.stdout: "begin": 2, 2024-06-03T05:30:55.647 INFO:tasks.workunit.client.0.smithi088.stdout: "length": 1 2024-06-03T05:30:55.647 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:30:55.647 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-06-03T05:30:55.647 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:30:55.647 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-06-03T05:30:55.647 INFO:tasks.workunit.client.0.smithi088.stdout: "new_removed_snaps": [ 2024-06-03T05:30:55.647 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.647 INFO:tasks.workunit.client.0.smithi088.stdout: "pool": 1, 2024-06-03T05:30:55.647 INFO:tasks.workunit.client.0.smithi088.stdout: "snaps": [ 2024-06-03T05:30:55.647 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.647 INFO:tasks.workunit.client.0.smithi088.stdout: "begin": 2, 2024-06-03T05:30:55.647 INFO:tasks.workunit.client.0.smithi088.stdout: "length": 1 2024-06-03T05:30:55.647 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:30:55.647 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-06-03T05:30:55.648 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:30:55.648 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-06-03T05:30:55.648 INFO:tasks.workunit.client.0.smithi088.stdout: "new_purged_snaps": [], 2024-06-03T05:30:55.648 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_node_flags": {}, 2024-06-03T05:30:55.648 INFO:tasks.workunit.client.0.smithi088.stdout: "device_class_flags": {}, 2024-06-03T05:30:55.648 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_mode": { 2024-06-03T05:30:55.648 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_mode_enabled": false, 2024-06-03T05:30:55.648 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_bucket_count": 0, 2024-06-03T05:30:55.648 INFO:tasks.workunit.client.0.smithi088.stdout: "degraded_stretch_mode": 0, 2024-06-03T05:30:55.648 INFO:tasks.workunit.client.0.smithi088.stdout: "recovering_stretch_mode": 0, 2024-06-03T05:30:55.648 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_mode_bucket": 0 2024-06-03T05:30:55.648 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:30:55.649 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.649 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_metadata": [ 2024-06-03T05:30:55.649 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.649 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-06-03T05:30:55.649 INFO:tasks.workunit.client.0.smithi088.stdout: "arch": "x86_64", 2024-06-03T05:30:55.649 INFO:tasks.workunit.client.0.smithi088.stdout: "back_addr": "[v2:127.0.0.1:6802/1160640607,v1:127.0.0.1:6803/1160640607]", 2024-06-03T05:30:55.649 INFO:tasks.workunit.client.0.smithi088.stdout: "back_iface": "", 2024-06-03T05:30:55.649 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs": "1", 2024-06-03T05:30:55.649 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_db": "0", 2024-06-03T05:30:55.649 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_wal": "0", 2024-06-03T05:30:55.649 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_single_shared_device": "1", 2024-06-03T05:30:55.649 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_allocation_from_file": "0", 2024-06-03T05:30:55.649 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_access_mode": "file", 2024-06-03T05:30:55.649 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_block_size": "4096", 2024-06-03T05:30:55.650 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_devices": "sda", 2024-06-03T05:30:55.650 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-06-03T05:30:55.650 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-06-03T05:30:55.650 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_path": "td/ceph-helpers/0/block", 2024-06-03T05:30:55.650 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_rotational": "1", 2024-06-03T05:30:55.650 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_size": "107374182400", 2024-06-03T05:30:55.650 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_support_discard": "0", 2024-06-03T05:30:55.650 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_type": "hdd", 2024-06-03T05:30:55.650 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_min_alloc_size": "4096", 2024-06-03T05:30:55.650 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_release": "squid", 2024-06-03T05:30:55.650 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version": "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)", 2024-06-03T05:30:55.650 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_short": "19.0.0-2946-g3d89accf", 2024-06-03T05:30:55.650 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_when_created": "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)", 2024-06-03T05:30:55.650 INFO:tasks.workunit.client.0.smithi088.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-06-03T05:30:55.651 INFO:tasks.workunit.client.0.smithi088.stdout: "created_at": "2024-06-03T05:30:14.287713Z", 2024-06-03T05:30:55.651 INFO:tasks.workunit.client.0.smithi088.stdout: "default_device_class": "hdd", 2024-06-03T05:30:55.651 INFO:tasks.workunit.client.0.smithi088.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHZ6", 2024-06-03T05:30:55.651 INFO:tasks.workunit.client.0.smithi088.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-06-03T05:30:55.651 INFO:tasks.workunit.client.0.smithi088.stdout: "devices": "sda", 2024-06-03T05:30:55.651 INFO:tasks.workunit.client.0.smithi088.stdout: "distro": "ubuntu", 2024-06-03T05:30:55.651 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-06-03T05:30:55.651 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_version": "22.04", 2024-06-03T05:30:55.651 INFO:tasks.workunit.client.0.smithi088.stdout: "front_addr": "[v2:127.0.0.1:6800/1160640607,v1:127.0.0.1:6801/1160640607]", 2024-06-03T05:30:55.651 INFO:tasks.workunit.client.0.smithi088.stdout: "front_iface": "", 2024-06-03T05:30:55.651 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_back_addr": "[v2:127.0.0.1:6806/1160640607,v1:127.0.0.1:6807/1160640607]", 2024-06-03T05:30:55.651 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_front_addr": "[v2:127.0.0.1:6804/1160640607,v1:127.0.0.1:6805/1160640607]", 2024-06-03T05:30:55.651 INFO:tasks.workunit.client.0.smithi088.stdout: "hostname": "smithi088", 2024-06-03T05:30:55.652 INFO:tasks.workunit.client.0.smithi088.stdout: "journal_rotational": "1", 2024-06-03T05:30:55.652 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_description": "#117-Ubuntu SMP Fri Apr 26 12:26:49 UTC 2024", 2024-06-03T05:30:55.652 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_version": "5.15.0-107-generic", 2024-06-03T05:30:55.652 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_swap_kb": "2097148", 2024-06-03T05:30:55.652 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_total_kb": "32746872", 2024-06-03T05:30:55.652 INFO:tasks.workunit.client.0.smithi088.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-06-03T05:30:55.652 INFO:tasks.workunit.client.0.smithi088.stdout: "objectstore_numa_unknown_devices": "sda", 2024-06-03T05:30:55.653 INFO:tasks.workunit.client.0.smithi088.stdout: "os": "Linux", 2024-06-03T05:30:55.653 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_data": "td/ceph-helpers/0", 2024-06-03T05:30:55.653 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_objectstore": "bluestore", 2024-06-03T05:30:55.653 INFO:tasks.workunit.client.0.smithi088.stdout: "osdspec_affinity": "", 2024-06-03T05:30:55.653 INFO:tasks.workunit.client.0.smithi088.stdout: "rotational": "1" 2024-06-03T05:30:55.653 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:30:55.653 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-06-03T05:30:55.653 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap_clean_epochs": { 2024-06-03T05:30:55.653 INFO:tasks.workunit.client.0.smithi088.stdout: "min_last_epoch_clean": 0, 2024-06-03T05:30:55.653 INFO:tasks.workunit.client.0.smithi088.stdout: "last_epoch_clean": { 2024-06-03T05:30:55.653 INFO:tasks.workunit.client.0.smithi088.stdout: "per_pool": [] 2024-06-03T05:30:55.653 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.653 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_epochs": [ 2024-06-03T05:30:55.653 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.654 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-06-03T05:30:55.654 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 5 2024-06-03T05:30:55.654 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:30:55.654 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-06-03T05:30:55.654 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.654 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap_first_committed": 1, 2024-06-03T05:30:55.654 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap_last_committed": 12, 2024-06-03T05:30:55.654 INFO:tasks.workunit.client.0.smithi088.stdout: "crushmap": { 2024-06-03T05:30:55.654 INFO:tasks.workunit.client.0.smithi088.stdout: "devices": [ 2024-06-03T05:30:55.654 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.654 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-06-03T05:30:55.654 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "osd.0", 2024-06-03T05:30:55.654 INFO:tasks.workunit.client.0.smithi088.stdout: "class": "hdd" 2024-06-03T05:30:55.655 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:30:55.655 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-06-03T05:30:55.655 INFO:tasks.workunit.client.0.smithi088.stdout: "types": [ 2024-06-03T05:30:55.655 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.655 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 0, 2024-06-03T05:30:55.655 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "osd" 2024-06-03T05:30:55.655 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.655 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.655 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 1, 2024-06-03T05:30:55.655 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "host" 2024-06-03T05:30:55.655 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.655 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.655 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 2, 2024-06-03T05:30:55.655 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "chassis" 2024-06-03T05:30:55.656 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.656 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.656 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 3, 2024-06-03T05:30:55.656 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "rack" 2024-06-03T05:30:55.656 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.656 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.656 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 4, 2024-06-03T05:30:55.656 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "row" 2024-06-03T05:30:55.656 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.656 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.656 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 5, 2024-06-03T05:30:55.656 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "pdu" 2024-06-03T05:30:55.656 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.656 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.656 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 6, 2024-06-03T05:30:55.657 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "pod" 2024-06-03T05:30:55.657 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.657 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.657 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 7, 2024-06-03T05:30:55.657 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "room" 2024-06-03T05:30:55.657 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.657 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.657 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 8, 2024-06-03T05:30:55.657 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "datacenter" 2024-06-03T05:30:55.657 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.657 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.657 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 9, 2024-06-03T05:30:55.657 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "zone" 2024-06-03T05:30:55.657 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.658 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.658 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 10, 2024-06-03T05:30:55.658 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "region" 2024-06-03T05:30:55.658 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.658 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.658 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 11, 2024-06-03T05:30:55.658 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "root" 2024-06-03T05:30:55.658 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:30:55.658 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-06-03T05:30:55.658 INFO:tasks.workunit.client.0.smithi088.stdout: "buckets": [ 2024-06-03T05:30:55.658 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.658 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -1, 2024-06-03T05:30:55.658 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "default", 2024-06-03T05:30:55.658 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 11, 2024-06-03T05:30:55.659 INFO:tasks.workunit.client.0.smithi088.stdout: "type_name": "root", 2024-06-03T05:30:55.659 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-06-03T05:30:55.659 INFO:tasks.workunit.client.0.smithi088.stdout: "alg": "straw2", 2024-06-03T05:30:55.659 INFO:tasks.workunit.client.0.smithi088.stdout: "hash": "rjenkins1", 2024-06-03T05:30:55.659 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-06-03T05:30:55.659 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.659 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -3, 2024-06-03T05:30:55.659 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-06-03T05:30:55.659 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 0 2024-06-03T05:30:55.659 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:30:55.659 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-06-03T05:30:55.659 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.659 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.659 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -2, 2024-06-03T05:30:55.659 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "default~hdd", 2024-06-03T05:30:55.660 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 11, 2024-06-03T05:30:55.660 INFO:tasks.workunit.client.0.smithi088.stdout: "type_name": "root", 2024-06-03T05:30:55.660 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-06-03T05:30:55.660 INFO:tasks.workunit.client.0.smithi088.stdout: "alg": "straw2", 2024-06-03T05:30:55.660 INFO:tasks.workunit.client.0.smithi088.stdout: "hash": "rjenkins1", 2024-06-03T05:30:55.660 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-06-03T05:30:55.660 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.660 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -4, 2024-06-03T05:30:55.660 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-06-03T05:30:55.660 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 0 2024-06-03T05:30:55.660 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:30:55.660 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-06-03T05:30:55.660 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.660 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.661 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -3, 2024-06-03T05:30:55.661 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "smithi088", 2024-06-03T05:30:55.661 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 1, 2024-06-03T05:30:55.661 INFO:tasks.workunit.client.0.smithi088.stdout: "type_name": "host", 2024-06-03T05:30:55.661 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-06-03T05:30:55.661 INFO:tasks.workunit.client.0.smithi088.stdout: "alg": "straw2", 2024-06-03T05:30:55.661 INFO:tasks.workunit.client.0.smithi088.stdout: "hash": "rjenkins1", 2024-06-03T05:30:55.661 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-06-03T05:30:55.661 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.661 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-06-03T05:30:55.661 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-06-03T05:30:55.661 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 0 2024-06-03T05:30:55.661 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:30:55.661 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-06-03T05:30:55.661 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.662 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.662 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -4, 2024-06-03T05:30:55.662 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "smithi088~hdd", 2024-06-03T05:30:55.662 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 1, 2024-06-03T05:30:55.662 INFO:tasks.workunit.client.0.smithi088.stdout: "type_name": "host", 2024-06-03T05:30:55.662 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-06-03T05:30:55.662 INFO:tasks.workunit.client.0.smithi088.stdout: "alg": "straw2", 2024-06-03T05:30:55.662 INFO:tasks.workunit.client.0.smithi088.stdout: "hash": "rjenkins1", 2024-06-03T05:30:55.662 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-06-03T05:30:55.662 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.662 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-06-03T05:30:55.662 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-06-03T05:30:55.662 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 0 2024-06-03T05:30:55.663 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:30:55.663 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-06-03T05:30:55.663 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:30:55.663 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-06-03T05:30:55.663 INFO:tasks.workunit.client.0.smithi088.stdout: "rules": [ 2024-06-03T05:30:55.663 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.663 INFO:tasks.workunit.client.0.smithi088.stdout: "rule_id": 0, 2024-06-03T05:30:55.663 INFO:tasks.workunit.client.0.smithi088.stdout: "rule_name": "replicated_rule", 2024-06-03T05:30:55.663 INFO:tasks.workunit.client.0.smithi088.stdout: "type": 1, 2024-06-03T05:30:55.663 INFO:tasks.workunit.client.0.smithi088.stdout: "steps": [ 2024-06-03T05:30:55.663 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.663 INFO:tasks.workunit.client.0.smithi088.stdout: "op": "take", 2024-06-03T05:30:55.663 INFO:tasks.workunit.client.0.smithi088.stdout: "item": -1, 2024-06-03T05:30:55.663 INFO:tasks.workunit.client.0.smithi088.stdout: "item_name": "default" 2024-06-03T05:30:55.663 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.664 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.664 INFO:tasks.workunit.client.0.smithi088.stdout: "op": "choose_firstn", 2024-06-03T05:30:55.664 INFO:tasks.workunit.client.0.smithi088.stdout: "num": 0, 2024-06-03T05:30:55.664 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "osd" 2024-06-03T05:30:55.664 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.664 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.664 INFO:tasks.workunit.client.0.smithi088.stdout: "op": "emit" 2024-06-03T05:30:55.664 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:30:55.664 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-06-03T05:30:55.664 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:30:55.664 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-06-03T05:30:55.664 INFO:tasks.workunit.client.0.smithi088.stdout: "tunables": { 2024-06-03T05:30:55.664 INFO:tasks.workunit.client.0.smithi088.stdout: "choose_local_tries": 0, 2024-06-03T05:30:55.664 INFO:tasks.workunit.client.0.smithi088.stdout: "choose_local_fallback_tries": 0, 2024-06-03T05:30:55.665 INFO:tasks.workunit.client.0.smithi088.stdout: "choose_total_tries": 50, 2024-06-03T05:30:55.665 INFO:tasks.workunit.client.0.smithi088.stdout: "chooseleaf_descend_once": 1, 2024-06-03T05:30:55.665 INFO:tasks.workunit.client.0.smithi088.stdout: "chooseleaf_vary_r": 1, 2024-06-03T05:30:55.665 INFO:tasks.workunit.client.0.smithi088.stdout: "chooseleaf_stable": 1, 2024-06-03T05:30:55.665 INFO:tasks.workunit.client.0.smithi088.stdout: "msr_descents": 100, 2024-06-03T05:30:55.665 INFO:tasks.workunit.client.0.smithi088.stdout: "msr_collision_tries": 100, 2024-06-03T05:30:55.665 INFO:tasks.workunit.client.0.smithi088.stdout: "straw_calc_version": 1, 2024-06-03T05:30:55.665 INFO:tasks.workunit.client.0.smithi088.stdout: "allowed_bucket_algs": 54, 2024-06-03T05:30:55.665 INFO:tasks.workunit.client.0.smithi088.stdout: "profile": "jewel", 2024-06-03T05:30:55.665 INFO:tasks.workunit.client.0.smithi088.stdout: "optimal_tunables": 1, 2024-06-03T05:30:55.665 INFO:tasks.workunit.client.0.smithi088.stdout: "legacy_tunables": 0, 2024-06-03T05:30:55.665 INFO:tasks.workunit.client.0.smithi088.stdout: "minimum_required_version": "jewel", 2024-06-03T05:30:55.665 INFO:tasks.workunit.client.0.smithi088.stdout: "require_feature_tunables": 1, 2024-06-03T05:30:55.665 INFO:tasks.workunit.client.0.smithi088.stdout: "require_feature_tunables2": 1, 2024-06-03T05:30:55.665 INFO:tasks.workunit.client.0.smithi088.stdout: "has_v2_rules": 0, 2024-06-03T05:30:55.666 INFO:tasks.workunit.client.0.smithi088.stdout: "require_feature_tunables3": 1, 2024-06-03T05:30:55.666 INFO:tasks.workunit.client.0.smithi088.stdout: "has_v3_rules": 0, 2024-06-03T05:30:55.666 INFO:tasks.workunit.client.0.smithi088.stdout: "has_v4_buckets": 1, 2024-06-03T05:30:55.666 INFO:tasks.workunit.client.0.smithi088.stdout: "require_feature_tunables5": 1, 2024-06-03T05:30:55.666 INFO:tasks.workunit.client.0.smithi088.stdout: "has_v5_rules": 0, 2024-06-03T05:30:55.666 INFO:tasks.workunit.client.0.smithi088.stdout: "has_msr_rules": 0 2024-06-03T05:30:55.666 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.666 INFO:tasks.workunit.client.0.smithi088.stdout: "choose_args": {} 2024-06-03T05:30:55.666 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.666 INFO:tasks.workunit.client.0.smithi088.stdout: "fsmap": { 2024-06-03T05:30:55.666 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 1, 2024-06-03T05:30:55.666 INFO:tasks.workunit.client.0.smithi088.stdout: "default_fscid": -1, 2024-06-03T05:30:55.666 INFO:tasks.workunit.client.0.smithi088.stdout: "compat": { 2024-06-03T05:30:55.667 INFO:tasks.workunit.client.0.smithi088.stdout: "compat": {}, 2024-06-03T05:30:55.667 INFO:tasks.workunit.client.0.smithi088.stdout: "ro_compat": {}, 2024-06-03T05:30:55.667 INFO:tasks.workunit.client.0.smithi088.stdout: "incompat": { 2024-06-03T05:30:55.667 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_1": "base v0.20", 2024-06-03T05:30:55.667 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_2": "client writeable ranges", 2024-06-03T05:30:55.667 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_3": "default file layouts on dirs", 2024-06-03T05:30:55.667 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_4": "dir inode in separate object", 2024-06-03T05:30:55.667 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_5": "mds uses versioned encoding", 2024-06-03T05:30:55.667 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_6": "dirfrag is stored in omap", 2024-06-03T05:30:55.667 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_8": "no anchor table", 2024-06-03T05:30:55.667 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_9": "file layout v2", 2024-06-03T05:30:55.667 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_10": "snaprealm v2", 2024-06-03T05:30:55.667 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_11": "minor log segments", 2024-06-03T05:30:55.667 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_12": "quiesce subvolumes" 2024-06-03T05:30:55.667 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:30:55.668 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.668 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_flags": { 2024-06-03T05:30:55.668 INFO:tasks.workunit.client.0.smithi088.stdout: "enable_multiple": true, 2024-06-03T05:30:55.668 INFO:tasks.workunit.client.0.smithi088.stdout: "ever_enabled_multiple": true 2024-06-03T05:30:55.668 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.668 INFO:tasks.workunit.client.0.smithi088.stdout: "standbys": [], 2024-06-03T05:30:55.668 INFO:tasks.workunit.client.0.smithi088.stdout: "filesystems": [] 2024-06-03T05:30:55.668 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.668 INFO:tasks.workunit.client.0.smithi088.stdout: "mdsmap_first_committed": 1, 2024-06-03T05:30:55.668 INFO:tasks.workunit.client.0.smithi088.stdout: "mdsmap_last_committed": 1, 2024-06-03T05:30:55.668 INFO:tasks.workunit.client.0.smithi088.stdout: "auth": { 2024-06-03T05:30:55.668 INFO:tasks.workunit.client.0.smithi088.stdout: "first_committed": 1, 2024-06-03T05:30:55.668 INFO:tasks.workunit.client.0.smithi088.stdout: "last_committed": 3, 2024-06-03T05:30:55.668 INFO:tasks.workunit.client.0.smithi088.stdout: "num_secrets": 1 2024-06-03T05:30:55.668 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.669 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg": 4, 2024-06-03T05:30:55.669 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg_active": 4, 2024-06-03T05:30:55.669 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg_unknown": 0, 2024-06-03T05:30:55.669 INFO:tasks.workunit.client.0.smithi088.stdout: "num_osd": 1, 2024-06-03T05:30:55.669 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_sum": { 2024-06-03T05:30:55.669 INFO:tasks.workunit.client.0.smithi088.stdout: "stat_sum": { 2024-06-03T05:30:55.669 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes": 19, 2024-06-03T05:30:55.669 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects": 1, 2024-06-03T05:30:55.669 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_clones": 0, 2024-06-03T05:30:55.669 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_copies": 2, 2024-06-03T05:30:55.669 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing_on_primary": 0, 2024-06-03T05:30:55.669 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing": 0, 2024-06-03T05:30:55.669 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_degraded": 1, 2024-06-03T05:30:55.669 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_misplaced": 0, 2024-06-03T05:30:55.670 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_unfound": 0, 2024-06-03T05:30:55.670 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_dirty": 1, 2024-06-03T05:30:55.670 INFO:tasks.workunit.client.0.smithi088.stdout: "num_whiteouts": 0, 2024-06-03T05:30:55.670 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read": 0, 2024-06-03T05:30:55.670 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read_kb": 0, 2024-06-03T05:30:55.670 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write": 2, 2024-06-03T05:30:55.670 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write_kb": 2, 2024-06-03T05:30:55.670 INFO:tasks.workunit.client.0.smithi088.stdout: "num_scrub_errors": 0, 2024-06-03T05:30:55.670 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shallow_scrub_errors": 0, 2024-06-03T05:30:55.670 INFO:tasks.workunit.client.0.smithi088.stdout: "num_deep_scrub_errors": 0, 2024-06-03T05:30:55.670 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_recovered": 0, 2024-06-03T05:30:55.670 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_recovered": 0, 2024-06-03T05:30:55.670 INFO:tasks.workunit.client.0.smithi088.stdout: "num_keys_recovered": 0, 2024-06-03T05:30:55.670 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_omap": 0, 2024-06-03T05:30:55.671 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_hit_set_archive": 0, 2024-06-03T05:30:55.671 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_hit_set_archive": 0, 2024-06-03T05:30:55.671 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush": 0, 2024-06-03T05:30:55.671 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_kb": 0, 2024-06-03T05:30:55.671 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict": 0, 2024-06-03T05:30:55.671 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_kb": 0, 2024-06-03T05:30:55.671 INFO:tasks.workunit.client.0.smithi088.stdout: "num_promote": 0, 2024-06-03T05:30:55.671 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_high": 0, 2024-06-03T05:30:55.671 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_low": 0, 2024-06-03T05:30:55.671 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_some": 0, 2024-06-03T05:30:55.671 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_full": 0, 2024-06-03T05:30:55.671 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_pinned": 0, 2024-06-03T05:30:55.671 INFO:tasks.workunit.client.0.smithi088.stdout: "num_legacy_snapsets": 0, 2024-06-03T05:30:55.671 INFO:tasks.workunit.client.0.smithi088.stdout: "num_large_omap_objects": 0, 2024-06-03T05:30:55.671 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_manifest": 0, 2024-06-03T05:30:55.672 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_bytes": 0, 2024-06-03T05:30:55.672 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_keys": 0, 2024-06-03T05:30:55.672 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_repaired": 0 2024-06-03T05:30:55.672 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.672 INFO:tasks.workunit.client.0.smithi088.stdout: "store_stats": { 2024-06-03T05:30:55.672 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 0, 2024-06-03T05:30:55.672 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 0, 2024-06-03T05:30:55.672 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-06-03T05:30:55.672 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 0, 2024-06-03T05:30:55.672 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 0, 2024-06-03T05:30:55.672 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-06-03T05:30:55.672 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-06-03T05:30:55.672 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-06-03T05:30:55.672 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 0, 2024-06-03T05:30:55.673 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 0 2024-06-03T05:30:55.673 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.673 INFO:tasks.workunit.client.0.smithi088.stdout: "log_size": 2, 2024-06-03T05:30:55.673 INFO:tasks.workunit.client.0.smithi088.stdout: "ondisk_log_size": 2, 2024-06-03T05:30:55.673 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 4, 2024-06-03T05:30:55.673 INFO:tasks.workunit.client.0.smithi088.stdout: "acting": 4, 2024-06-03T05:30:55.673 INFO:tasks.workunit.client.0.smithi088.stdout: "num_store_stats": 0 2024-06-03T05:30:55.673 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.673 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_sum": { 2024-06-03T05:30:55.673 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 0, 2024-06-03T05:30:55.673 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 0, 2024-06-03T05:30:55.673 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pgs": 4, 2024-06-03T05:30:55.673 INFO:tasks.workunit.client.0.smithi088.stdout: "num_osds": 1, 2024-06-03T05:30:55.673 INFO:tasks.workunit.client.0.smithi088.stdout: "num_per_pool_osds": 1, 2024-06-03T05:30:55.674 INFO:tasks.workunit.client.0.smithi088.stdout: "num_per_pool_omap_osds": 1, 2024-06-03T05:30:55.674 INFO:tasks.workunit.client.0.smithi088.stdout: "kb": 104857600, 2024-06-03T05:30:55.674 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used": 26948, 2024-06-03T05:30:55.674 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_data": 108, 2024-06-03T05:30:55.674 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_omap": 1, 2024-06-03T05:30:55.674 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_meta": 26814, 2024-06-03T05:30:55.674 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_avail": 104830652, 2024-06-03T05:30:55.674 INFO:tasks.workunit.client.0.smithi088.stdout: "statfs": { 2024-06-03T05:30:55.674 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 107374182400, 2024-06-03T05:30:55.674 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 107346587648, 2024-06-03T05:30:55.674 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-06-03T05:30:55.674 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 110592, 2024-06-03T05:30:55.674 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 22938, 2024-06-03T05:30:55.674 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-06-03T05:30:55.675 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-06-03T05:30:55.675 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-06-03T05:30:55.675 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 1595, 2024-06-03T05:30:55.675 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 27457989 2024-06-03T05:30:55.675 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.675 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_peers": [], 2024-06-03T05:30:55.675 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_trim_queue_len": 0, 2024-06-03T05:30:55.675 INFO:tasks.workunit.client.0.smithi088.stdout: "num_snap_trimming": 0, 2024-06-03T05:30:55.675 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shards_repaired": 0, 2024-06-03T05:30:55.675 INFO:tasks.workunit.client.0.smithi088.stdout: "op_queue_age_hist": { 2024-06-03T05:30:55.675 INFO:tasks.workunit.client.0.smithi088.stdout: "histogram": [], 2024-06-03T05:30:55.675 INFO:tasks.workunit.client.0.smithi088.stdout: "upper_bound": 1 2024-06-03T05:30:55.675 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.675 INFO:tasks.workunit.client.0.smithi088.stdout: "perf_stat": { 2024-06-03T05:30:55.675 INFO:tasks.workunit.client.0.smithi088.stdout: "commit_latency_ms": 0, 2024-06-03T05:30:55.676 INFO:tasks.workunit.client.0.smithi088.stdout: "apply_latency_ms": 0, 2024-06-03T05:30:55.676 INFO:tasks.workunit.client.0.smithi088.stdout: "commit_latency_ns": 0, 2024-06-03T05:30:55.676 INFO:tasks.workunit.client.0.smithi088.stdout: "apply_latency_ns": 0 2024-06-03T05:30:55.676 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.676 INFO:tasks.workunit.client.0.smithi088.stdout: "alerts": [], 2024-06-03T05:30:55.676 INFO:tasks.workunit.client.0.smithi088.stdout: "network_ping_times": [] 2024-06-03T05:30:55.676 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.676 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_sum_by_class": { 2024-06-03T05:30:55.676 INFO:tasks.workunit.client.0.smithi088.stdout: "hdd": { 2024-06-03T05:30:55.676 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 0, 2024-06-03T05:30:55.676 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 0, 2024-06-03T05:30:55.676 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pgs": 4, 2024-06-03T05:30:55.676 INFO:tasks.workunit.client.0.smithi088.stdout: "num_osds": 1, 2024-06-03T05:30:55.676 INFO:tasks.workunit.client.0.smithi088.stdout: "num_per_pool_osds": 1, 2024-06-03T05:30:55.676 INFO:tasks.workunit.client.0.smithi088.stdout: "num_per_pool_omap_osds": 1, 2024-06-03T05:30:55.676 INFO:tasks.workunit.client.0.smithi088.stdout: "kb": 104857600, 2024-06-03T05:30:55.677 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used": 26948, 2024-06-03T05:30:55.677 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_data": 108, 2024-06-03T05:30:55.677 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_omap": 1, 2024-06-03T05:30:55.677 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_meta": 26814, 2024-06-03T05:30:55.677 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_avail": 104830652, 2024-06-03T05:30:55.677 INFO:tasks.workunit.client.0.smithi088.stdout: "statfs": { 2024-06-03T05:30:55.677 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 107374182400, 2024-06-03T05:30:55.677 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 107346587648, 2024-06-03T05:30:55.677 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-06-03T05:30:55.677 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 110592, 2024-06-03T05:30:55.677 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 22938, 2024-06-03T05:30:55.677 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-06-03T05:30:55.677 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-06-03T05:30:55.677 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-06-03T05:30:55.678 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 1595, 2024-06-03T05:30:55.678 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 27457989 2024-06-03T05:30:55.678 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.678 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_peers": [], 2024-06-03T05:30:55.678 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_trim_queue_len": 0, 2024-06-03T05:30:55.678 INFO:tasks.workunit.client.0.smithi088.stdout: "num_snap_trimming": 0, 2024-06-03T05:30:55.678 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shards_repaired": 0, 2024-06-03T05:30:55.678 INFO:tasks.workunit.client.0.smithi088.stdout: "op_queue_age_hist": { 2024-06-03T05:30:55.678 INFO:tasks.workunit.client.0.smithi088.stdout: "histogram": [], 2024-06-03T05:30:55.678 INFO:tasks.workunit.client.0.smithi088.stdout: "upper_bound": 1 2024-06-03T05:30:55.678 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.678 INFO:tasks.workunit.client.0.smithi088.stdout: "perf_stat": { 2024-06-03T05:30:55.678 INFO:tasks.workunit.client.0.smithi088.stdout: "commit_latency_ms": 0, 2024-06-03T05:30:55.678 INFO:tasks.workunit.client.0.smithi088.stdout: "apply_latency_ms": 0, 2024-06-03T05:30:55.679 INFO:tasks.workunit.client.0.smithi088.stdout: "commit_latency_ns": 0, 2024-06-03T05:30:55.679 INFO:tasks.workunit.client.0.smithi088.stdout: "apply_latency_ns": 0 2024-06-03T05:30:55.679 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.679 INFO:tasks.workunit.client.0.smithi088.stdout: "alerts": [], 2024-06-03T05:30:55.679 INFO:tasks.workunit.client.0.smithi088.stdout: "network_ping_times": [] 2024-06-03T05:30:55.679 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:30:55.679 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.679 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_stats": [ 2024-06-03T05:30:55.679 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.679 INFO:tasks.workunit.client.0.smithi088.stdout: "poolid": 1, 2024-06-03T05:30:55.679 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg": 4, 2024-06-03T05:30:55.679 INFO:tasks.workunit.client.0.smithi088.stdout: "stat_sum": { 2024-06-03T05:30:55.679 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes": 19, 2024-06-03T05:30:55.679 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects": 1, 2024-06-03T05:30:55.680 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_clones": 0, 2024-06-03T05:30:55.680 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_copies": 2, 2024-06-03T05:30:55.680 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing_on_primary": 0, 2024-06-03T05:30:55.680 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing": 0, 2024-06-03T05:30:55.680 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_degraded": 1, 2024-06-03T05:30:55.680 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_misplaced": 0, 2024-06-03T05:30:55.680 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_unfound": 0, 2024-06-03T05:30:55.680 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_dirty": 1, 2024-06-03T05:30:55.680 INFO:tasks.workunit.client.0.smithi088.stdout: "num_whiteouts": 0, 2024-06-03T05:30:55.680 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read": 0, 2024-06-03T05:30:55.680 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read_kb": 0, 2024-06-03T05:30:55.680 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write": 2, 2024-06-03T05:30:55.680 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write_kb": 2, 2024-06-03T05:30:55.680 INFO:tasks.workunit.client.0.smithi088.stdout: "num_scrub_errors": 0, 2024-06-03T05:30:55.681 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shallow_scrub_errors": 0, 2024-06-03T05:30:55.681 INFO:tasks.workunit.client.0.smithi088.stdout: "num_deep_scrub_errors": 0, 2024-06-03T05:30:55.681 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_recovered": 0, 2024-06-03T05:30:55.681 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_recovered": 0, 2024-06-03T05:30:55.681 INFO:tasks.workunit.client.0.smithi088.stdout: "num_keys_recovered": 0, 2024-06-03T05:30:55.681 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_omap": 0, 2024-06-03T05:30:55.681 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_hit_set_archive": 0, 2024-06-03T05:30:55.681 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_hit_set_archive": 0, 2024-06-03T05:30:55.681 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush": 0, 2024-06-03T05:30:55.681 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_kb": 0, 2024-06-03T05:30:55.681 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict": 0, 2024-06-03T05:30:55.681 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_kb": 0, 2024-06-03T05:30:55.681 INFO:tasks.workunit.client.0.smithi088.stdout: "num_promote": 0, 2024-06-03T05:30:55.681 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_high": 0, 2024-06-03T05:30:55.682 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_low": 0, 2024-06-03T05:30:55.682 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_some": 0, 2024-06-03T05:30:55.682 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_full": 0, 2024-06-03T05:30:55.682 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_pinned": 0, 2024-06-03T05:30:55.682 INFO:tasks.workunit.client.0.smithi088.stdout: "num_legacy_snapsets": 0, 2024-06-03T05:30:55.682 INFO:tasks.workunit.client.0.smithi088.stdout: "num_large_omap_objects": 0, 2024-06-03T05:30:55.682 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_manifest": 0, 2024-06-03T05:30:55.682 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_bytes": 0, 2024-06-03T05:30:55.682 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_keys": 0, 2024-06-03T05:30:55.682 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_repaired": 0 2024-06-03T05:30:55.682 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.682 INFO:tasks.workunit.client.0.smithi088.stdout: "store_stats": { 2024-06-03T05:30:55.682 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 0, 2024-06-03T05:30:55.682 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 0, 2024-06-03T05:30:55.683 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-06-03T05:30:55.683 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 4096, 2024-06-03T05:30:55.683 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 19, 2024-06-03T05:30:55.683 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-06-03T05:30:55.683 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-06-03T05:30:55.683 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-06-03T05:30:55.683 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 0, 2024-06-03T05:30:55.683 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 0 2024-06-03T05:30:55.683 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.683 INFO:tasks.workunit.client.0.smithi088.stdout: "log_size": 2, 2024-06-03T05:30:55.683 INFO:tasks.workunit.client.0.smithi088.stdout: "ondisk_log_size": 2, 2024-06-03T05:30:55.683 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 4, 2024-06-03T05:30:55.683 INFO:tasks.workunit.client.0.smithi088.stdout: "acting": 4, 2024-06-03T05:30:55.683 INFO:tasks.workunit.client.0.smithi088.stdout: "num_store_stats": 1 2024-06-03T05:30:55.684 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:30:55.684 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-06-03T05:30:55.684 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_stats": [ 2024-06-03T05:30:55.684 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.684 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 0, 2024-06-03T05:30:55.684 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 21474836482 2024-06-03T05:30:55.684 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:30:55.684 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-06-03T05:30:55.684 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg_by_state": [ 2024-06-03T05:30:55.684 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.684 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "active+undersized+degraded", 2024-06-03T05:30:55.684 INFO:tasks.workunit.client.0.smithi088.stdout: "num": 1 2024-06-03T05:30:55.684 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.684 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.684 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "active+undersized", 2024-06-03T05:30:55.685 INFO:tasks.workunit.client.0.smithi088.stdout: "num": 3 2024-06-03T05:30:55.685 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:30:55.685 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-06-03T05:30:55.685 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg_by_osd": [ 2024-06-03T05:30:55.685 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.685 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 0, 2024-06-03T05:30:55.685 INFO:tasks.workunit.client.0.smithi088.stdout: "num_primary_pg": 4, 2024-06-03T05:30:55.685 INFO:tasks.workunit.client.0.smithi088.stdout: "num_acting_pg": 4, 2024-06-03T05:30:55.685 INFO:tasks.workunit.client.0.smithi088.stdout: "num_up_not_acting_pg": 0 2024-06-03T05:30:55.685 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:30:55.685 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-06-03T05:30:55.685 INFO:tasks.workunit.client.0.smithi088.stdout: "purged_snaps": [ 2024-06-03T05:30:55.685 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-06-03T05:30:55.685 INFO:tasks.workunit.client.0.smithi088.stdout: "pool": 1, 2024-06-03T05:30:55.685 INFO:tasks.workunit.client.0.smithi088.stdout: "purged_snaps": {} 2024-06-03T05:30:55.686 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:30:55.686 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-06-03T05:30:55.686 INFO:tasks.workunit.client.0.smithi088.stdout: "servicemap": { 2024-06-03T05:30:55.686 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 1, 2024-06-03T05:30:55.686 INFO:tasks.workunit.client.0.smithi088.stdout: "modified": "2024-06-03T05:30:07.818849+0000", 2024-06-03T05:30:55.686 INFO:tasks.workunit.client.0.smithi088.stdout: "services": {} 2024-06-03T05:30:55.686 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-06-03T05:30:55.686 INFO:tasks.workunit.client.0.smithi088.stdout: "mgrstat_first_committed": 1, 2024-06-03T05:30:55.686 INFO:tasks.workunit.client.0.smithi088.stdout: "mgrstat_last_committed": 6, 2024-06-03T05:30:55.686 INFO:tasks.workunit.client.0.smithi088.stdout: "logm_first_committed": 1, 2024-06-03T05:30:55.686 INFO:tasks.workunit.client.0.smithi088.stdout: "logm_last_committed": 77, 2024-06-03T05:30:55.686 INFO:tasks.workunit.client.0.smithi088.stdout: "paxos": { 2024-06-03T05:30:55.686 INFO:tasks.workunit.client.0.smithi088.stdout: "first_committed": 1, 2024-06-03T05:30:55.687 INFO:tasks.workunit.client.0.smithi088.stdout: "last_committed": 102, 2024-06-03T05:30:55.687 INFO:tasks.workunit.client.0.smithi088.stdout: "last_pn": 0, 2024-06-03T05:30:55.687 INFO:tasks.workunit.client.0.smithi088.stdout: "accepted_pn": 0 2024-06-03T05:30:55.687 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-06-03T05:30:55.687 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-06-03T05:30:55.690 INFO:tasks.workunit.client.0.smithi088.stderr:report 2118961779 2024-06-03T05:30:55.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-06-03T05:30:55.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1742: test_wait_for_peered: run_osd td/ceph-helpers 1 2024-06-03T05:30:55.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:30:55.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:30:55.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-03T05:30:55.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:30:55.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-06-03T05:30:55.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:30:55.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:30:55.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:30:55.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:30:55.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-06-03T05:30:55.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-06-03T05:30:55.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:30:55.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:30:55.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:30:55.691 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:30:55.691 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:30:55.691 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:30:55.692 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:30:55.692 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:30:55.692 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:30:55.692 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:30:55.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:30:55.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:30:55.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:30:55.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:30:55.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:30:55.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:30:55.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:30:55.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:30:55.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:30:55.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:30:55.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:30:55.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:30:55.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-06-03T05:30:55.693 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:30:55.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d9c844ed-2108-4ce0-b15e-10735172e3ea 2024-06-03T05:30:55.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 d9c844ed-2108-4ce0-b15e-10735172e3ea' 2024-06-03T05:30:55.693 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 d9c844ed-2108-4ce0-b15e-10735172e3ea 2024-06-03T05:30:55.694 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:30:55.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAPVV1mkPRVJxAATZjLPN6oLMQmTMsNAZCf8A== 2024-06-03T05:30:55.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAPVV1mkPRVJxAATZjLPN6oLMQmTMsNAZCf8A=="}' 2024-06-03T05:30:55.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d9c844ed-2108-4ce0-b15e-10735172e3ea -i td/ceph-helpers/1/new.json 2024-06-03T05:30:56.145 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:30:56.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-06-03T05:30:56.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAPVV1mkPRVJxAATZjLPN6oLMQmTMsNAZCf8A== --osd-uuid d9c844ed-2108-4ce0-b15e-10735172e3ea 2024-06-03T05:30:56.198 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:30:56.190+0000 7f73536ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:30:56.223 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:30:56.218+0000 7f73536ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:30:56.224 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:30:56.218+0000 7f73536ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:30:56.225 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:30:56.218+0000 7f73536ec7c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-03T05:30:56.225 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:30:56.218+0000 7f73536ec7c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-03T05:30:56.225 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:30:56.218+0000 7f73536ec7c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-03T05:30:56.271 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:30:56.266+0000 7f73536ec7c0 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-06-03T05:31:02.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-06-03T05:31:02.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:31:02.975 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-06-03T05:31:02.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-03T05:31:02.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:31:03.528 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-06-03T05:31:03.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-03T05:31:03.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:31:03.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:31:03.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:31:03.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:31:03.559 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:31:03.554+0000 7fc3668e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:31:03.596 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:31:03.590+0000 7fc3668e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:31:03.597 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:31:03.590+0000 7fc3668e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:31:03.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-03T05:31:03.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:31:03.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T05:31:03.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:31:03.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:31:03.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:31:03.952 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:31:03.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:31:03.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:31:03.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:31:04.152 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:31:04.146+0000 7fc3668e37c0 -1 Falling back to public interface 2024-06-03T05:31:04.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:31:05.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:31:05.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:31:05.366 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:31:05.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:31:05.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:31:05.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:31:05.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:31:06.739 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:31:06.734+0000 7fc3668e37c0 -1 osd.1 0 log_to_monitors true 2024-06-03T05:31:06.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:31:06.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:31:06.777 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:31:06.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:31:06.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:31:06.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:31:07.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:31:08.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:31:08.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:31:08.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:31:08.284 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:31:08.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:31:08.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:31:08.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:31:09.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:31:09.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:31:09.799 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:31:09.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:31:09.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:31:09.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:31:10.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:31:11.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:31:11.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:31:11.219 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:31:11.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:31:11.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:31:11.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:31:11.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:31:12.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:31:12.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:31:12.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:31:12.834 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:31:12.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:31:12.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:31:13.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:31:14.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:31:14.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:31:14.434 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:31:14.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:31:14.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:31:14.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:31:14.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:31:15.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:31:15.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:31:15.916 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:31:15.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:31:15.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:31:15.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:31:16.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:31:17.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:31:17.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:31:17.348 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:31:17.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:31:17.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:31:17.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:31:17.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:31:18.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:31:18.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:31:18.971 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:31:18.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:31:18.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:31:18.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:31:19.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:31:20.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:31:20.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:31:20.531 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:31:20.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:31:20.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:31:20.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:31:21.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:31:22.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:31:22.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:31:22.462 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:31:22.462 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:31:22.462 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:31:22.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:31:23.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:31:24.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:31:24.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:31:24.651 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:31:24.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:31:24.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:31:24.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:31:25.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:31:26.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:31:26.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:31:26.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:31:26.840 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:31:26.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:31:26.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:31:27.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:31:28.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:31:28.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:31:28.956 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:31:28.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:31:28.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:31:28.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:31:29.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:31:30.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:31:30.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:31:30.886 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:31:30.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:31:30.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:31:30.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:31:32.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:31:33.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:31:33.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:31:33.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:31:33.099 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:31:33.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:31:33.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:31:33.617 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 16 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4045957063,v1:127.0.0.1:6811/4045957063] [v2:127.0.0.1:6812/4045957063,v1:127.0.0.1:6813/4045957063] exists,up d9c844ed-2108-4ce0-b15e-10735172e3ea 2024-06-03T05:31:33.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:31:33.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:31:33.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:31:33.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1743: test_wait_for_peered: wait_for_peered 2024-06-03T05:31:33.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: wait_for_peered: local cmd= 2024-06-03T05:31:33.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: wait_for_peered: local num_peered=-1 2024-06-03T05:31:33.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1701: wait_for_peered: local cur_peered 2024-06-03T05:31:33.619 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: get_timeout_delays 90 .1 2024-06-03T05:31:33.619 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:31:33.620 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:31:33.620 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:31:33.620 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:31:33.620 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:31:33.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T05:31:33.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: local -a delays 2024-06-03T05:31:33.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_peered: local -i loop=0 2024-06-03T05:31:33.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1705: wait_for_peered: flush_pg_stats 2024-06-03T05:31:33.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T05:31:33.716 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T05:31:34.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T05:31:34.217 INFO:tasks.workunit.client.0.smithi088.stderr:1' 2024-06-03T05:31:34.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T05:31:34.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:31:34.217 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T05:31:34.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836492 2024-06-03T05:31:34.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836492 2024-06-03T05:31:34.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836492' 2024-06-03T05:31:34.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:31:34.376 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T05:31:34.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=68719476742 2024-06-03T05:31:34.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 68719476742 2024-06-03T05:31:34.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836492 1-68719476742' 2024-06-03T05:31:34.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:31:34.533 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836492 2024-06-03T05:31:34.533 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:31:34.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T05:31:34.534 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836492 2024-06-03T05:31:34.534 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:31:34.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836492 2024-06-03T05:31:34.535 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836492 2024-06-03T05:31:34.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-06-03T05:31:34.536 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:31:35.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836492 2024-06-03T05:31:35.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:31:35.002 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-68719476742 2024-06-03T05:31:35.003 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:31:35.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T05:31:35.005 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-68719476742 2024-06-03T05:31:35.006 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:31:35.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=68719476742 2024-06-03T05:31:35.007 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 68719476742 2024-06-03T05:31:35.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 68719476742' 2024-06-03T05:31:35.007 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T05:31:35.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 68719476742 -lt 68719476742 2024-06-03T05:31:35.436 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: get_num_pgs 2024-06-03T05:31:35.437 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:31:35.437 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:31:36.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: test 4 == 0 2024-06-03T05:31:36.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_peered: true 2024-06-03T05:31:36.032 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: get_num_active_or_peered 2024-06-03T05:31:36.032 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1394: get_num_active_or_peered: local expression 2024-06-03T05:31:36.032 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2024-06-03T05:31:36.032 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2024-06-03T05:31:36.032 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: get_num_active_or_peered: ceph --format json pg dump pgs 2024-06-03T05:31:36.032 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2024-06-03T05:31:36.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: cur_peered=4 2024-06-03T05:31:36.438 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: get_num_pgs 2024-06-03T05:31:36.439 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:31:36.439 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:31:36.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: test 4 = 4 2024-06-03T05:31:36.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: break 2024-06-03T05:31:36.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1730: wait_for_peered: return 0 2024-06-03T05:31:36.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1744: test_wait_for_peered: teardown td/ceph-helpers 2024-06-03T05:31:36.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:31:36.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:31:36.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:31:36.958 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:31:36.958 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:31:36.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:31:36.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:31:36.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:31:37.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:31:37.267 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:31:37.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:31:37.268 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:31:37.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:31:37.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:31:37.270 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:31:37.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:31:37.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:31:37.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:31:37.272 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:31:37.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:31:37.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:31:37.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:31:37.288 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:31:37.288 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:31:37.288 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:31:37.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:31:37.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:31:37.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:31:37.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:31:37.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_wait_for_scrub td/ceph-helpers 2024-06-03T05:31:37.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: test_wait_for_scrub: local dir=td/ceph-helpers 2024-06-03T05:31:37.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: test_wait_for_scrub: setup td/ceph-helpers 2024-06-03T05:31:37.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-03T05:31:37.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-03T05:31:37.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:31:37.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:31:37.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:31:37.291 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:31:37.291 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:31:37.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:31:37.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:31:37.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:31:37.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:31:37.293 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:31:37.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:31:37.294 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:31:37.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:31:37.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:31:37.295 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:31:37.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:31:37.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:31:37.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:31:37.297 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:31:37.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:31:37.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:31:37.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:31:37.300 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:31:37.301 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:31:37.301 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:31:37.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:31:37.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:31:37.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:31:37.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-03T05:31:37.303 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:31:37.303 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:31:37.303 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:31:37.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70897 2024-06-03T05:31:37.306 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:31:37.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:31:37.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:31:37.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-03T05:31:37.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: test_wait_for_scrub: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:31:37.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-03T05:31:37.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:31:37.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:31:37.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:31:37.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-03T05:31:37.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:31:37.742 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:31:37.742 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:31:37.742 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:31:37.743 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:31:37.743 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:31:37.743 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:31:37.743 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:31:37.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:31:37.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:31:37.961 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:31:37.961 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:31:37.961 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:31:37.961 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:31:37.962 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:31:37.962 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:31:37.962 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:31:37.962 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:31:37.962 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:31:37.963 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:31:37.963 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:31:37.963 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:31:37.963 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:31:37.963 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get fsid 2024-06-03T05:31:38.078 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:31:38.078 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:31:38.079 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:31:38.079 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:31:38.079 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:31:38.080 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:31:38.080 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:31:38.080 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:31:38.080 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:31:38.080 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:31:38.080 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:31:38.080 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:31:38.080 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:31:38.080 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get mon_host 2024-06-03T05:31:38.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: test_wait_for_scrub: run_mgr td/ceph-helpers x 2024-06-03T05:31:38.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-03T05:31:38.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T05:31:38.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T05:31:38.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T05:31:38.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-03T05:31:38.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T05:31:39.083 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T05:31:39.083 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:31:39.083 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:31:39.083 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:31:39.083 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:31:39.083 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:31:39.083 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:31:39.084 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:31:39.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:31:39.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: test_wait_for_scrub: run_osd td/ceph-helpers 0 2024-06-03T05:31:39.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:31:39.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:31:39.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T05:31:39.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:31:39.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-03T05:31:39.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:31:39.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:31:39.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:31:39.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:31:39.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-03T05:31:39.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-03T05:31:39.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:31:39.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:31:39.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:31:39.123 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:31:39.123 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:31:39.123 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:31:39.123 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:31:39.123 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:31:39.123 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:31:39.123 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:31:39.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:31:39.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:31:39.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:31:39.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:31:39.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:31:39.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:31:39.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:31:39.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:31:39.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:31:39.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:31:39.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:31:39.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:31:39.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-03T05:31:39.125 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:31:39.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f84f3620-5e9f-423f-bfd1-1359634eb84d 2024-06-03T05:31:39.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f84f3620-5e9f-423f-bfd1-1359634eb84d' 2024-06-03T05:31:39.125 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 f84f3620-5e9f-423f-bfd1-1359634eb84d 2024-06-03T05:31:39.125 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:31:39.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA7VV1mJrkDCBAABqgqAFX3N4U9Yc5rd1aRxw== 2024-06-03T05:31:39.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA7VV1mJrkDCBAABqgqAFX3N4U9Yc5rd1aRxw=="}' 2024-06-03T05:31:39.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f84f3620-5e9f-423f-bfd1-1359634eb84d -i td/ceph-helpers/0/new.json 2024-06-03T05:31:39.449 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:31:39.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-03T05:31:39.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA7VV1mJrkDCBAABqgqAFX3N4U9Yc5rd1aRxw== --osd-uuid f84f3620-5e9f-423f-bfd1-1359634eb84d 2024-06-03T05:31:39.499 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:31:39.494+0000 7fcf515d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:31:39.530 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:31:39.522+0000 7fcf515d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:31:39.531 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:31:39.522+0000 7fcf515d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:31:39.531 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:31:39.522+0000 7fcf515d87c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:31:39.531 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:31:39.522+0000 7fcf515d87c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:31:39.531 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:31:39.522+0000 7fcf515d87c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:31:39.566 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:31:39.562+0000 7fcf515d87c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-03T05:31:45.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-03T05:31:45.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:31:45.872 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T05:31:45.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T05:31:45.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:31:46.426 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:31:46.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T05:31:46.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:31:46.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:31:46.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:31:46.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:31:46.459 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:31:46.454+0000 7f745eab97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:31:46.484 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:31:46.478+0000 7f745eab97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:31:46.485 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:31:46.478+0000 7f745eab97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:31:46.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T05:31:46.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:31:46.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:31:46.846 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:31:46.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:31:46.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:31:46.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:31:46.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:31:46.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:31:46.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:31:47.068 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:31:47.062+0000 7f745eab97c0 -1 Falling back to public interface 2024-06-03T05:31:47.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:31:48.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:31:48.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:31:48.261 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:31:48.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:31:48.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:31:48.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:31:48.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:31:49.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:31:49.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:31:49.688 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:31:49.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:31:49.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:31:49.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:31:49.691 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:31:49.686+0000 7f745eab97c0 -1 osd.0 0 log_to_monitors true 2024-06-03T05:31:50.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:31:51.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:31:51.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:31:51.171 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:31:51.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:31:51.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:31:51.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:31:51.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:31:52.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:31:52.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:31:52.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:31:52.648 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:31:52.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:31:52.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:31:53.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:31:54.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:31:54.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:31:54.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:31:54.071 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:31:54.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:31:54.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:31:54.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:31:55.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:31:55.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:31:55.619 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:31:55.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:31:55.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:31:55.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:31:56.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:31:57.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:31:57.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:31:57.057 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:31:57.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:31:57.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:31:57.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:31:57.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:31:58.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:31:58.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:31:58.687 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:31:58.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:31:58.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:31:58.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:31:59.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:32:00.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:32:00.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:32:00.111 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:32:00.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:32:00.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:32:00.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:32:00.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:32:01.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:32:01.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:32:01.619 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:32:01.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:32:01.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:32:01.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:32:02.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:32:03.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:32:03.044 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:32:03.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:32:03.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:32:03.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:32:03.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:32:03.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:32:04.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:32:04.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:32:04.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:32:04.517 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:32:04.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:32:04.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:32:05.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:32:06.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:32:06.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:32:06.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:32:06.552 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:32:06.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:32:06.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:32:07.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:32:08.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:32:08.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:32:08.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:32:08.467 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:32:08.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:32:08.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:32:09.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:32:10.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:32:10.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:32:10.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:32:10.403 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:32:10.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:32:10.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:32:11.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:32:12.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:32:12.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:32:12.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:32:12.477 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:32:12.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:32:12.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:32:13.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:32:14.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:32:14.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:32:14.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:32:14.292 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:32:14.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:32:14.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:32:14.689 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1957319247,v1:127.0.0.1:6803/1957319247] [v2:127.0.0.1:6804/1957319247,v1:127.0.0.1:6805/1957319247] exists,up f84f3620-5e9f-423f-bfd1-1359634eb84d 2024-06-03T05:32:14.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:32:14.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:32:14.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:32:14.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: test_wait_for_scrub: create_rbd_pool 2024-06-03T05:32:14.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-03T05:32:15.094 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' does not exist 2024-06-03T05:32:15.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-03T05:32:15.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-03T05:32:15.622 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' already exists 2024-06-03T05:32:15.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T05:32:16.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-03T05:32:18.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: test_wait_for_scrub: wait_for_clean 2024-06-03T05:32:18.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T05:32:18.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T05:32:18.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T05:32:18.011 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T05:32:18.012 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:32:18.012 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:32:18.013 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:32:18.013 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:32:18.013 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:32:18.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T05:32:18.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T05:32:18.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T05:32:18.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T05:32:18.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T05:32:18.133 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T05:32:18.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-06-03T05:32:18.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T05:32:18.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:32:18.565 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T05:32:18.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836487 2024-06-03T05:32:18.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836487 2024-06-03T05:32:18.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836487' 2024-06-03T05:32:18.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:32:18.734 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836487 2024-06-03T05:32:18.734 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:32:18.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T05:32:18.735 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836487 2024-06-03T05:32:18.735 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:32:18.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836487 2024-06-03T05:32:18.736 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836487 2024-06-03T05:32:18.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-06-03T05:32:18.737 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:32:19.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836485 -lt 21474836487 2024-06-03T05:32:19.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T05:32:20.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T05:32:20.156 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:32:20.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836487 -lt 21474836487 2024-06-03T05:32:20.559 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T05:32:20.559 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:32:20.559 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:32:21.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-03T05:32:21.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T05:32:21.146 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T05:32:21.146 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T05:32:21.146 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T05:32:21.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T05:32:21.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T05:32:21.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T05:32:21.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T05:32:21.527 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T05:32:21.527 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:32:21.528 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:32:22.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-03T05:32:22.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T05:32:22.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T05:32:22.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: test_wait_for_scrub: local pgid=1.0 2024-06-03T05:32:22.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2032: test_wait_for_scrub: ceph pg repair 1.0 2024-06-03T05:32:22.425 INFO:tasks.workunit.client.0.smithi088.stderr:instructing pg 1.0 on osd.0 to repair 2024-06-03T05:32:22.441 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: test_wait_for_scrub: get_last_scrub_stamp 1.0 2024-06-03T05:32:22.441 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T05:32:22.441 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T05:32:22.442 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T05:32:22.442 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T05:32:22.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: test_wait_for_scrub: local last_scrub=2024-06-03T05:32:15.468682+0000 2024-06-03T05:32:22.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2034: test_wait_for_scrub: wait_for_scrub 1.0 2024-06-03T05:32:15.468682+0000 2024-06-03T05:32:22.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-06-03T05:32:22.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T05:32:15.468682+0000 2024-06-03T05:32:22.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T05:32:22.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T05:32:22.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T05:32:22.830 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T05:32:22.830 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T05:32:22.830 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T05:32:22.830 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T05:32:22.830 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T05:32:23.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T05:32:15.468682+0000 '>' 2024-06-03T05:32:15.468682+0000 2024-06-03T05:32:23.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T05:32:24.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T05:32:24.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T05:32:24.208 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T05:32:24.208 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T05:32:24.208 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T05:32:24.209 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T05:32:24.209 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T05:32:24.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T05:32:15.468682+0000 '>' 2024-06-03T05:32:15.468682+0000 2024-06-03T05:32:24.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T05:32:25.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T05:32:25.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-03T05:32:25.609 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T05:32:25.609 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T05:32:25.609 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T05:32:25.610 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T05:32:25.610 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T05:32:26.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T05:32:22.853553+0000 '>' 2024-06-03T05:32:15.468682+0000 2024-06-03T05:32:26.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-03T05:32:26.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2035: test_wait_for_scrub: kill_daemons td/ceph-helpers KILL osd 2024-06-03T05:32:26.028 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:32:26.028 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:32:26.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:32:26.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:32:26.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:32:26.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:32:26.138 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2036: test_wait_for_scrub: get_last_scrub_stamp 1.0 2024-06-03T05:32:26.138 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T05:32:26.138 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T05:32:26.138 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T05:32:26.139 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T05:32:26.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2036: test_wait_for_scrub: last_scrub=2024-06-03T05:32:22.853553+0000 2024-06-03T05:32:26.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2037: test_wait_for_scrub: TIMEOUT=1 2024-06-03T05:32:26.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2037: test_wait_for_scrub: wait_for_scrub 1.0 2024-06-03T05:32:22.853553+0000 2024-06-03T05:32:26.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-06-03T05:32:26.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-03T05:32:22.853553+0000 2024-06-03T05:32:26.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-03T05:32:26.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-03T05:32:26.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 1 )) 2024-06-03T05:32:26.567 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-03T05:32:26.567 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-03T05:32:26.567 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-03T05:32:26.568 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-03T05:32:26.568 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-03T05:32:26.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-03T05:32:22.853553+0000 '>' 2024-06-03T05:32:22.853553+0000 2024-06-03T05:32:26.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-03T05:32:27.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-03T05:32:27.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 1 )) 2024-06-03T05:32:27.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: return 1 2024-06-03T05:32:27.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2038: test_wait_for_scrub: teardown td/ceph-helpers 2024-06-03T05:32:27.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:32:27.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:32:27.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:32:27.948 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:32:27.948 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:32:27.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:32:27.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:32:27.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:32:28.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:32:28.058 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:32:28.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:32:28.059 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:32:28.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:32:28.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:32:28.061 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:32:28.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:32:28.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:32:28.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:32:28.064 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:32:28.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:32:28.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:32:28.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:32:28.074 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:32:28.074 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:32:28.074 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:32:28.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:32:28.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:32:28.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:32:28.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-06-03T05:32:28.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_wait_osd_id_used_by_pgs td/ceph-helpers 2024-06-03T05:32:28.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1501: test_wait_osd_id_used_by_pgs: local dir=td/ceph-helpers 2024-06-03T05:32:28.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1503: test_wait_osd_id_used_by_pgs: setup td/ceph-helpers 2024-06-03T05:32:28.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-03T05:32:28.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-03T05:32:28.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:32:28.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:32:28.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:32:28.077 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:32:28.077 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:32:28.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:32:28.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:32:28.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:32:28.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:32:28.079 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:32:28.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:32:28.081 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:32:28.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:32:28.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:32:28.082 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:32:28.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:32:28.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:32:28.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:32:28.084 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:32:28.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:32:28.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:32:28.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:32:28.086 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:32:28.087 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:32:28.087 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:32:28.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:32:28.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:32:28.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:32:28.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-03T05:32:28.089 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:32:28.089 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:32:28.089 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:32:28.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70897 2024-06-03T05:32:28.090 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:32:28.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:32:28.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:32:28.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-03T05:32:28.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1504: test_wait_osd_id_used_by_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:32:28.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-03T05:32:28.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:32:28.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:32:28.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:32:28.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-03T05:32:28.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:32:28.496 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:32:28.496 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:32:28.497 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:32:28.497 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:32:28.497 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:32:28.497 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:32:28.497 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:32:28.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-03T05:32:28.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:32:28.710 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:32:28.710 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:32:28.710 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:32:28.710 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:32:28.710 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:32:28.711 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:32:28.711 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:32:28.711 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:32:28.711 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:32:28.712 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:32:28.712 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:32:28.712 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:32:28.712 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:32:28.712 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get fsid 2024-06-03T05:32:28.830 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:32:28.830 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:32:28.830 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:32:28.830 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:32:28.831 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:32:28.831 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:32:28.831 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:32:28.831 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:32:28.832 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:32:28.832 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:32:28.832 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:32:28.832 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70897/ceph-mon.a.asok 2024-06-03T05:32:28.832 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:32:28.832 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70897/ceph-mon.a.asok config get mon_host 2024-06-03T05:32:28.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1505: test_wait_osd_id_used_by_pgs: run_mgr td/ceph-helpers x 2024-06-03T05:32:28.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-03T05:32:28.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T05:32:28.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T05:32:28.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T05:32:28.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-03T05:32:28.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T05:32:29.856 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T05:32:29.856 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:32:29.856 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:32:29.858 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:32:29.858 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:32:29.858 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:32:29.858 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:32:29.858 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:32:29.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:32:29.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1506: test_wait_osd_id_used_by_pgs: run_osd td/ceph-helpers 0 2024-06-03T05:32:29.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-03T05:32:29.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:32:29.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T05:32:29.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:32:29.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-03T05:32:29.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-03T05:32:29.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:32:29.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:32:29.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:32:29.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-03T05:32:29.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-03T05:32:29.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:32:29.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:32:29.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-03T05:32:29.898 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:32:29.898 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:32:29.898 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:32:29.898 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:32:29.898 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:32:29.898 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:32:29.898 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:32:29.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' 2024-06-03T05:32:29.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:32:29.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:32:29.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:32:29.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-03T05:32:29.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-03T05:32:29.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:32:29.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:32:29.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:32:29.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:32:29.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:32:29.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:32:29.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-03T05:32:29.899 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:32:29.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=92778adb-0c85-4fc5-b5dd-707f34f6cf28 2024-06-03T05:32:29.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 92778adb-0c85-4fc5-b5dd-707f34f6cf28' 2024-06-03T05:32:29.899 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 92778adb-0c85-4fc5-b5dd-707f34f6cf28 2024-06-03T05:32:29.900 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:32:29.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBtVV1mqnVeNhAA5ymdGIMPApDVv++2kTKTOQ== 2024-06-03T05:32:29.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBtVV1mqnVeNhAA5ymdGIMPApDVv++2kTKTOQ=="}' 2024-06-03T05:32:29.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 92778adb-0c85-4fc5-b5dd-707f34f6cf28 -i td/ceph-helpers/0/new.json 2024-06-03T05:32:30.212 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:32:30.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-03T05:32:30.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBtVV1mqnVeNhAA5ymdGIMPApDVv++2kTKTOQ== --osd-uuid 92778adb-0c85-4fc5-b5dd-707f34f6cf28 2024-06-03T05:32:30.267 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:32:30.262+0000 7fe45f25e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:32:30.280 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:32:30.274+0000 7fe45f25e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:32:30.282 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:32:30.274+0000 7fe45f25e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:32:30.282 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:32:30.278+0000 7fe45f25e7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:32:30.283 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:32:30.278+0000 7fe45f25e7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:32:30.283 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:32:30.278+0000 7fe45f25e7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-03T05:32:30.321 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:32:30.314+0000 7fe45f25e7c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-03T05:32:36.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-03T05:32:36.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:32:36.517 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T05:32:36.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T05:32:36.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:32:37.121 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:32:37.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T05:32:37.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=213fb4bd-379a-40f7-a383-f6c59451eb9c --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:32:37.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:32:37.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:32:37.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:32:37.154 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:32:37.146+0000 7f4dd1f817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:32:37.179 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:32:37.174+0000 7f4dd1f817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:32:37.180 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:32:37.174+0000 7f4dd1f817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:32:37.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T05:32:37.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:32:37.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:32:37.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:32:37.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:32:37.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:32:37.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:32:37.542 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:32:37.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:32:37.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:32:37.759 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:32:37.754+0000 7f4dd1f817c0 -1 Falling back to public interface 2024-06-03T05:32:37.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:32:38.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:32:38.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:32:38.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:32:38.943 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:32:38.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:32:38.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:32:39.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:32:40.353 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:32:40.346+0000 7f4dd1f817c0 -1 osd.0 0 log_to_monitors true 2024-06-03T05:32:40.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:32:40.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:32:40.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:32:40.354 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:32:40.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:32:40.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:32:40.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:32:41.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:32:41.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:32:41.824 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:32:41.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:32:41.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:32:41.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:32:42.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:32:43.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:32:43.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:32:43.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:32:43.381 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:32:43.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:32:43.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:32:43.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:32:44.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:32:44.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:32:44.869 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:32:44.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:32:44.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:32:44.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:32:45.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:32:46.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:32:46.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:32:46.344 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:32:46.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:32:46.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:32:46.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:32:46.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:32:47.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:32:47.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:32:47.775 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:32:47.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:32:47.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:32:47.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:32:48.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:32:49.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:32:49.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:32:49.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:32:49.290 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:32:49.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:32:49.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:32:49.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:32:50.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:32:50.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:32:50.701 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:32:50.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:32:50.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:32:50.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:32:51.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:32:52.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:32:52.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:32:52.247 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:32:52.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:32:52.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:32:52.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:32:52.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:32:53.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:32:53.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:32:53.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:32:53.700 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:32:53.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:32:53.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:32:54.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:32:55.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:32:55.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:32:55.148 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:32:55.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:32:55.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:32:55.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:32:56.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:32:57.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:32:57.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:32:57.184 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:32:57.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:32:57.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:32:57.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:32:58.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:32:59.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:32:59.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:32:59.003 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:32:59.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:32:59.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:32:59.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:33:00.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:33:01.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:33:01.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:33:01.221 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:33:01.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:33:01.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:33:01.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:33:02.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:33:03.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:33:03.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:33:03.415 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:33:03.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:33:03.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:33:03.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:33:04.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:33:05.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:33:05.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:33:05.191 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:33:05.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:33:05.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:33:05.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:33:05.623 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4288520041,v1:127.0.0.1:6803/4288520041] [v2:127.0.0.1:6804/4288520041,v1:127.0.0.1:6805/4288520041] exists,up 92778adb-0c85-4fc5-b5dd-707f34f6cf28 2024-06-03T05:33:05.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:33:05.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:33:05.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:33:05.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1507: test_wait_osd_id_used_by_pgs: create_rbd_pool 2024-06-03T05:33:05.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-03T05:33:06.036 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' does not exist 2024-06-03T05:33:06.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-03T05:33:06.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-03T05:33:06.594 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' already exists 2024-06-03T05:33:06.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T05:33:07.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-03T05:33:08.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1508: test_wait_osd_id_used_by_pgs: wait_for_clean 2024-06-03T05:33:08.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T05:33:08.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T05:33:08.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T05:33:08.754 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T05:33:08.754 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:33:08.754 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:33:08.754 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:33:08.754 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:33:08.755 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:33:08.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T05:33:08.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T05:33:08.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T05:33:08.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T05:33:08.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T05:33:08.875 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T05:33:09.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-06-03T05:33:09.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T05:33:09.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:33:09.287 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T05:33:09.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836485 2024-06-03T05:33:09.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836485 2024-06-03T05:33:09.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836485' 2024-06-03T05:33:09.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:33:09.439 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836485 2024-06-03T05:33:09.440 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:33:09.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T05:33:09.441 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836485 2024-06-03T05:33:09.441 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:33:09.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836485 2024-06-03T05:33:09.442 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836485 2024-06-03T05:33:09.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2024-06-03T05:33:09.442 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:33:09.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836483 -lt 21474836485 2024-06-03T05:33:09.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T05:33:10.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T05:33:10.842 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:33:11.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836485 -lt 21474836485 2024-06-03T05:33:11.282 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T05:33:11.282 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:33:11.283 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:33:11.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-03T05:33:11.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T05:33:11.821 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T05:33:11.822 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T05:33:11.822 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T05:33:11.822 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T05:33:11.822 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T05:33:11.822 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T05:33:12.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-03T05:33:12.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T05:33:12.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:33:12.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:33:12.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-03T05:33:12.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T05:33:12.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T05:33:12.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1509: test_wait_osd_id_used_by_pgs: wait_osd_id_used_by_pgs 0 8 2024-06-03T05:33:12.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1484: wait_osd_id_used_by_pgs: local id=0 2024-06-03T05:33:12.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1485: wait_osd_id_used_by_pgs: local count=8 2024-06-03T05:33:12.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1487: wait_osd_id_used_by_pgs: status=1 2024-06-03T05:33:12.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1488: wait_osd_id_used_by_pgs: (( i=0 )) 2024-06-03T05:33:12.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1488: wait_osd_id_used_by_pgs: (( i < 300 / 5 )) 2024-06-03T05:33:12.798 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:33:12.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1489: wait_osd_id_used_by_pgs: echo 0 2024-06-03T05:33:12.799 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1490: wait_osd_id_used_by_pgs: get_osd_id_used_by_pgs 2024-06-03T05:33:12.799 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1490: wait_osd_id_used_by_pgs: grep -c 0 2024-06-03T05:33:12.799 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: ceph --format json pg dump pgs 2024-06-03T05:33:12.799 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: jq '.pg_stats | .[] | .up[], .acting[]' 2024-06-03T05:33:12.799 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: sort 2024-06-03T05:33:13.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1490: wait_osd_id_used_by_pgs: test 8 = 8 2024-06-03T05:33:13.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1493: wait_osd_id_used_by_pgs: status=0 2024-06-03T05:33:13.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: wait_osd_id_used_by_pgs: break 2024-06-03T05:33:13.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1497: wait_osd_id_used_by_pgs: return 0 2024-06-03T05:33:13.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1510: test_wait_osd_id_used_by_pgs: TIMEOUT=1 2024-06-03T05:33:13.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1510: test_wait_osd_id_used_by_pgs: wait_osd_id_used_by_pgs 123 5 2024-06-03T05:33:13.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1484: wait_osd_id_used_by_pgs: local id=123 2024-06-03T05:33:13.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1485: wait_osd_id_used_by_pgs: local count=5 2024-06-03T05:33:13.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1487: wait_osd_id_used_by_pgs: status=1 2024-06-03T05:33:13.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1488: wait_osd_id_used_by_pgs: (( i=0 )) 2024-06-03T05:33:13.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1488: wait_osd_id_used_by_pgs: (( i < 1 / 5 )) 2024-06-03T05:33:13.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1497: wait_osd_id_used_by_pgs: return 1 2024-06-03T05:33:13.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1511: test_wait_osd_id_used_by_pgs: teardown td/ceph-helpers 2024-06-03T05:33:13.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-03T05:33:13.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:33:13.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-03T05:33:13.181 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:33:13.181 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:33:13.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:33:13.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:33:13.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:33:13.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:33:13.489 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:33:13.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:33:13.491 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:33:13.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:33:13.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:33:13.493 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:33:13.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:33:13.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:33:13.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:33:13.495 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:33:13.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:33:13.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:33:13.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-03T05:33:13.505 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:33:13.505 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:33:13.505 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70897 2024-06-03T05:33:13.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70897 2024-06-03T05:33:13.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:33:13.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:33:13.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: : exit 0 2024-06-03T05:33:13.508 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-06-03T05:33:13.508 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-06-03T05:33:13.521 INFO:tasks.workunit:Running workunit misc/test-snaptrim-stats.sh... 2024-06-03T05:33:13.521 DEBUG:teuthology.orchestra.run.smithi088:workunit test misc/test-snaptrim-stats.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dfcc4b3c2c55532c4c04fa47551c9df1dffdf746 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh 2024-06-03T05:33:13.573 INFO:tasks.workunit.client.0.smithi088.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-06-03T05:33:13.578 INFO:tasks.workunit.client.0.smithi088.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-06-03T05:33:13.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-03T05:33:13.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-03T05:33:13.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-06-03T05:33:13.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-06-03T05:33:13.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-06-03T05:33:13.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-06-03T05:33:13.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-06-03T05:33:13.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-06-03T05:33:13.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/test-snaptrim-stats 2024-06-03T05:33:13.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:21: run: local dir=td/test-snaptrim-stats 2024-06-03T05:33:13.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:22: run: shift 2024-06-03T05:33:13.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:24: run: export CEPH_MON=127.0.0.1:7124 2024-06-03T05:33:13.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:24: run: CEPH_MON=127.0.0.1:7124 2024-06-03T05:33:13.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:25: run: export CEPH_ARGS 2024-06-03T05:33:13.579 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:26: run: uuidgen 2024-06-03T05:33:13.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:26: run: CEPH_ARGS+='--fsid=3e288c27-dfcb-4963-a1c0-21f4dfb9cbc8 --auth-supported=none ' 2024-06-03T05:33:13.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2024-06-03T05:33:13.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:28: run: CEPH_ARGS+='--debug-bluestore 20 ' 2024-06-03T05:33:13.580 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:30: run: set 2024-06-03T05:33:13.580 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-06-03T05:33:13.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:30: run: local 'funcs=TEST_snaptrim_stats 2024-06-03T05:33:13.582 INFO:tasks.workunit.client.0.smithi088.stderr:TEST_snaptrim_stats_multiple_snaps' 2024-06-03T05:33:13.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:31: run: for func in $funcs 2024-06-03T05:33:13.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:32: run: setup td/test-snaptrim-stats 2024-06-03T05:33:13.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-snaptrim-stats 2024-06-03T05:33:13.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-snaptrim-stats 2024-06-03T05:33:13.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-06-03T05:33:13.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:33:13.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-06-03T05:33:13.583 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:33:13.583 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:33:13.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:33:13.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:33:13.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:33:13.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:33:13.585 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:33:13.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:33:13.586 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:33:13.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:33:13.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:33:13.587 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:33:13.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:33:13.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:33:13.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:33:13.588 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:33:13.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:33:13.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:33:13.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-06-03T05:33:13.591 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:33:13.592 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:33:13.592 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169257 2024-06-03T05:33:13.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.169257 2024-06-03T05:33:13.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:33:13.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:33:13.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-snaptrim-stats 2024-06-03T05:33:13.594 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:33:13.594 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:33:13.594 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169257 2024-06-03T05:33:13.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.169257 2024-06-03T05:33:13.597 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:33:13.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-06-03T05:33:13.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-06-03T05:33:13.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:33:13.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-snaptrim-stats 1' TERM HUP INT 2024-06-03T05:33:13.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:33: run: TEST_snaptrim_stats td/test-snaptrim-stats 2024-06-03T05:33:13.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:39: TEST_snaptrim_stats: local dir=td/test-snaptrim-stats 2024-06-03T05:33:13.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:40: TEST_snaptrim_stats: local poolname=test 2024-06-03T05:33:13.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:41: TEST_snaptrim_stats: local OSDS=3 2024-06-03T05:33:13.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:42: TEST_snaptrim_stats: local PGNUM=8 2024-06-03T05:33:13.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:43: TEST_snaptrim_stats: local PGPNUM=8 2024-06-03T05:33:13.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:44: TEST_snaptrim_stats: local objects=10 2024-06-03T05:33:13.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:45: TEST_snaptrim_stats: local WAIT_FOR_UPDATE=10 2024-06-03T05:33:13.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:47: TEST_snaptrim_stats: setup td/test-snaptrim-stats 2024-06-03T05:33:13.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-snaptrim-stats 2024-06-03T05:33:13.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-snaptrim-stats 2024-06-03T05:33:13.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-06-03T05:33:13.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:33:13.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-06-03T05:33:13.599 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:33:13.599 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:33:13.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:33:13.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:33:13.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:33:13.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:33:13.600 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:33:13.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:33:13.600 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:33:13.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:33:13.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:33:13.601 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:33:13.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:33:13.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:33:13.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:33:13.603 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:33:13.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:33:13.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:33:13.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-06-03T05:33:13.606 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:33:13.606 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:33:13.606 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169257 2024-06-03T05:33:13.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.169257 2024-06-03T05:33:13.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:33:13.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:33:13.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-snaptrim-stats 2024-06-03T05:33:13.608 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:33:13.608 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:33:13.608 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169257 2024-06-03T05:33:13.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.169257 2024-06-03T05:33:13.611 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:33:13.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:33:13.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:33:13.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-snaptrim-stats 1' TERM HUP INT 2024-06-03T05:33:13.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:48: TEST_snaptrim_stats: run_mon td/test-snaptrim-stats a --osd_pool_default_size=3 2024-06-03T05:33:13.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-snaptrim-stats 2024-06-03T05:33:13.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:33:13.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:33:13.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:33:13.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-snaptrim-stats/a 2024-06-03T05:33:13.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-snaptrim-stats/a --run-dir=td/test-snaptrim-stats --osd_pool_default_size=3 2024-06-03T05:33:14.006 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:33:14.007 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:33:14.007 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:33:14.007 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:33:14.007 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:33:14.008 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169257 2024-06-03T05:33:14.008 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.169257/$cluster-$name.asok' 2024-06-03T05:33:14.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-snaptrim-stats/a '--log-file=td/test-snaptrim-stats/$name.log' '--admin-socket=/tmp/ceph-asok.169257/$cluster-$name.asok' --mon-cluster-log-file=td/test-snaptrim-stats/log --run-dir=td/test-snaptrim-stats '--pid-file=td/test-snaptrim-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-06-03T05:33:14.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:33:14.245 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:33:14.245 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:33:14.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:33:14.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:33:14.248 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:33:14.248 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:33:14.248 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:33:14.248 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:33:14.248 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:33:14.248 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:33:14.248 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169257 2024-06-03T05:33:14.248 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.169257/ceph-mon.a.asok 2024-06-03T05:33:14.248 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:33:14.248 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.169257/ceph-mon.a.asok config get fsid 2024-06-03T05:33:14.357 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:33:14.357 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:33:14.357 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:33:14.357 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:33:14.358 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:33:14.358 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:33:14.358 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:33:14.358 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:33:14.359 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:33:14.359 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:33:14.359 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169257 2024-06-03T05:33:14.359 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.169257/ceph-mon.a.asok 2024-06-03T05:33:14.359 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:33:14.359 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.169257/ceph-mon.a.asok config get mon_host 2024-06-03T05:33:14.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:49: TEST_snaptrim_stats: run_mgr td/test-snaptrim-stats x 2024-06-03T05:33:14.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-snaptrim-stats 2024-06-03T05:33:14.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T05:33:14.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T05:33:14.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T05:33:14.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-snaptrim-stats/x 2024-06-03T05:33:14.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T05:33:15.363 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T05:33:15.364 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:33:15.364 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:33:15.364 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:33:15.364 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:33:15.364 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169257 2024-06-03T05:33:15.364 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.169257/$cluster-$name.asok' 2024-06-03T05:33:15.365 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:33:15.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-snaptrim-stats/x '--log-file=td/test-snaptrim-stats/$name.log' '--admin-socket=/tmp/ceph-asok.169257/$cluster-$name.asok' --run-dir=td/test-snaptrim-stats '--pid-file=td/test-snaptrim-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:33:15.400 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: expr 3 - 1 2024-06-03T05:33:15.401 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: seq 0 2 2024-06-03T05:33:15.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T05:33:15.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:52: TEST_snaptrim_stats: run_osd td/test-snaptrim-stats 0 --osd_pool_default_pg_autoscale_mode=off 2024-06-03T05:33:15.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-06-03T05:33:15.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:33:15.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T05:33:15.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:33:15.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/0 2024-06-03T05:33:15.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3e288c27-dfcb-4963-a1c0-21f4dfb9cbc8 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-06-03T05:33:15.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:33:15.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:33:15.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:33:15.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/0' 2024-06-03T05:33:15.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/0/journal' 2024-06-03T05:33:15.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:33:15.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:33:15.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-06-03T05:33:15.407 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:33:15.407 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:33:15.407 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:33:15.408 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:33:15.408 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:33:15.408 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169257 2024-06-03T05:33:15.408 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.169257/$cluster-$name.asok' 2024-06-03T05:33:15.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.169257/$cluster-$name.asok' 2024-06-03T05:33:15.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:33:15.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:33:15.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:33:15.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-06-03T05:33:15.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-06-03T05:33:15.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:33:15.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:33:15.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:33:15.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:33:15.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:33:15.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-06-03T05:33:15.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/0 2024-06-03T05:33:15.409 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:33:15.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ebc584f3-8f12-4cd3-808c-530d7d154910 2024-06-03T05:33:15.409 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 ebc584f3-8f12-4cd3-808c-530d7d154910 2024-06-03T05:33:15.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ebc584f3-8f12-4cd3-808c-530d7d154910' 2024-06-03T05:33:15.410 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:33:15.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCbVV1mBDX/GBAA4GtY4DF/punqJKTJlhyy0w== 2024-06-03T05:33:15.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCbVV1mBDX/GBAA4GtY4DF/punqJKTJlhyy0w=="}' 2024-06-03T05:33:15.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ebc584f3-8f12-4cd3-808c-530d7d154910 -i td/test-snaptrim-stats/0/new.json 2024-06-03T05:33:15.739 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:33:15.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/0/new.json 2024-06-03T05:33:15.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=3e288c27-dfcb-4963-a1c0-21f4dfb9cbc8 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/0 --osd-journal=td/test-snaptrim-stats/0/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.169257/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQCbVV1mBDX/GBAA4GtY4DF/punqJKTJlhyy0w== --osd-uuid ebc584f3-8f12-4cd3-808c-530d7d154910 2024-06-03T05:33:15.788 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:33:15.782+0000 7f28eec907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:33:15.824 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:33:15.818+0000 7f28eec907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:33:15.826 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:33:15.818+0000 7f28eec907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:33:15.826 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:33:15.822+0000 7f28eec907c0 -1 bluestore(td/test-snaptrim-stats/0/block) _read_bdev_label failed to open td/test-snaptrim-stats/0/block: (2) No such file or directory 2024-06-03T05:33:15.827 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:33:15.822+0000 7f28eec907c0 -1 bluestore(td/test-snaptrim-stats/0/block) _read_bdev_label failed to open td/test-snaptrim-stats/0/block: (2) No such file or directory 2024-06-03T05:33:15.827 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:33:15.822+0000 7f28eec907c0 -1 bluestore(td/test-snaptrim-stats/0/block) _read_bdev_label failed to open td/test-snaptrim-stats/0/block: (2) No such file or directory 2024-06-03T05:33:15.873 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:33:15.866+0000 7f28eec907c0 -1 bluestore(td/test-snaptrim-stats/0) _read_fsid unparsable uuid 2024-06-03T05:33:22.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/0/keyring 2024-06-03T05:33:22.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:33:22.233 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T05:33:22.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T05:33:22.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:33:22.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T05:33:22.802 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:33:22.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=3e288c27-dfcb-4963-a1c0-21f4dfb9cbc8 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/0 --osd-journal=td/test-snaptrim-stats/0/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.169257/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-06-03T05:33:22.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:33:22.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:33:22.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:33:22.831 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:33:22.826+0000 7fd1da0f07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:33:22.866 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:33:22.862+0000 7fd1da0f07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:33:22.867 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:33:22.862+0000 7fd1da0f07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:33:23.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T05:33:23.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:33:23.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:33:23.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:33:23.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:33:23.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:33:23.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:33:23.183 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:33:23.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:33:23.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:33:23.421 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:33:23.414+0000 7fd1da0f07c0 -1 Falling back to public interface 2024-06-03T05:33:23.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:33:24.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:33:24.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:33:24.618 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:33:24.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:33:24.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:33:24.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:33:25.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:33:25.982 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:33:25.974+0000 7fd1da0f07c0 -1 osd.0 0 log_to_monitors true 2024-06-03T05:33:26.015 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:33:26.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:33:26.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:33:26.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:33:26.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:33:26.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:33:26.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:33:27.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:33:27.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:33:27.728 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:33:27.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:33:27.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:33:27.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:33:28.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:33:29.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:33:29.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:33:29.160 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:33:29.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:33:29.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:33:29.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:33:29.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:33:30.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:33:30.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:33:30.580 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:33:30.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:33:30.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:33:30.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:33:31.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:33:32.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:33:32.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:33:32.130 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:33:32.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:33:32.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:33:32.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:33:32.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:33:33.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:33:33.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:33:33.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:33:33.581 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:33:33.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:33:33.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:33:34.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:33:35.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:33:35.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:33:35.184 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:33:35.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:33:35.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:33:35.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:33:35.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:33:36.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:33:36.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:33:36.648 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:33:36.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:33:36.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:33:36.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:33:37.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:33:38.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:33:38.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:33:38.157 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:33:38.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:33:38.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:33:38.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:33:38.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:33:39.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:33:39.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:33:39.594 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:33:39.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:33:39.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:33:39.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:33:40.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:33:41.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:33:41.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:33:41.499 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:33:41.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:33:41.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:33:41.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:33:42.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:33:43.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:33:43.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:33:43.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:33:43.292 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:33:43.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:33:43.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:33:44.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:33:45.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:33:45.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:33:45.211 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:33:45.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:33:45.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:33:45.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:33:45.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:33:46.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:33:46.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:33:46.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:33:46.806 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:33:46.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:33:46.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:33:47.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:33:48.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:33:48.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:33:48.449 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:33:48.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:33:48.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:33:48.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:33:49.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:33:50.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:33:50.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:33:50.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:33:50.350 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:33:50.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:33:50.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:33:50.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:33:51.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:33:51.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:33:51.887 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-06-03T05:33:51.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T05:33:51.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:33:51.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:33:52.324 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4165520976,v1:127.0.0.1:6803/4165520976] [v2:127.0.0.1:6804/4165520976,v1:127.0.0.1:6805/4165520976] exists,up ebc584f3-8f12-4cd3-808c-530d7d154910 2024-06-03T05:33:52.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:33:52.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:33:52.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:33:52.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T05:33:52.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:52: TEST_snaptrim_stats: run_osd td/test-snaptrim-stats 1 --osd_pool_default_pg_autoscale_mode=off 2024-06-03T05:33:52.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-06-03T05:33:52.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:33:52.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-03T05:33:52.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:33:52.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/1 2024-06-03T05:33:52.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3e288c27-dfcb-4963-a1c0-21f4dfb9cbc8 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-06-03T05:33:52.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:33:52.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:33:52.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:33:52.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/1' 2024-06-03T05:33:52.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/1/journal' 2024-06-03T05:33:52.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:33:52.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:33:52.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-06-03T05:33:52.329 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:33:52.329 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:33:52.330 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:33:52.330 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:33:52.330 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:33:52.330 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169257 2024-06-03T05:33:52.330 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.169257/$cluster-$name.asok' 2024-06-03T05:33:52.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.169257/$cluster-$name.asok' 2024-06-03T05:33:52.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:33:52.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:33:52.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:33:52.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-06-03T05:33:52.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-06-03T05:33:52.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:33:52.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:33:52.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:33:52.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:33:52.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:33:52.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-06-03T05:33:52.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/1 2024-06-03T05:33:52.331 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:33:52.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=89a4f496-bfd0-4196-93ee-690da357b620 2024-06-03T05:33:52.332 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 89a4f496-bfd0-4196-93ee-690da357b620 2024-06-03T05:33:52.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 89a4f496-bfd0-4196-93ee-690da357b620' 2024-06-03T05:33:52.332 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:33:52.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDAVV1mXzrTFBAAyYrIxFTxuH6gHMNnlKbChA== 2024-06-03T05:33:52.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDAVV1mXzrTFBAAyYrIxFTxuH6gHMNnlKbChA=="}' 2024-06-03T05:33:52.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 89a4f496-bfd0-4196-93ee-690da357b620 -i td/test-snaptrim-stats/1/new.json 2024-06-03T05:33:52.840 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:33:52.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/1/new.json 2024-06-03T05:33:52.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=3e288c27-dfcb-4963-a1c0-21f4dfb9cbc8 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/1 --osd-journal=td/test-snaptrim-stats/1/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.169257/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQDAVV1mXzrTFBAAyYrIxFTxuH6gHMNnlKbChA== --osd-uuid 89a4f496-bfd0-4196-93ee-690da357b620 2024-06-03T05:33:52.891 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:33:52.886+0000 7fa45aecf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:33:52.908 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:33:52.902+0000 7fa45aecf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:33:52.910 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:33:52.902+0000 7fa45aecf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:33:52.910 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:33:52.906+0000 7fa45aecf7c0 -1 bluestore(td/test-snaptrim-stats/1/block) _read_bdev_label failed to open td/test-snaptrim-stats/1/block: (2) No such file or directory 2024-06-03T05:33:52.910 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:33:52.906+0000 7fa45aecf7c0 -1 bluestore(td/test-snaptrim-stats/1/block) _read_bdev_label failed to open td/test-snaptrim-stats/1/block: (2) No such file or directory 2024-06-03T05:33:52.911 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:33:52.906+0000 7fa45aecf7c0 -1 bluestore(td/test-snaptrim-stats/1/block) _read_bdev_label failed to open td/test-snaptrim-stats/1/block: (2) No such file or directory 2024-06-03T05:33:52.956 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:33:52.950+0000 7fa45aecf7c0 -1 bluestore(td/test-snaptrim-stats/1) _read_fsid unparsable uuid 2024-06-03T05:33:59.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/1/keyring 2024-06-03T05:33:59.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:33:59.705 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-06-03T05:33:59.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-03T05:33:59.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:34:00.258 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-06-03T05:34:00.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-03T05:34:00.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:34:00.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:34:00.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=3e288c27-dfcb-4963-a1c0-21f4dfb9cbc8 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/1 --osd-journal=td/test-snaptrim-stats/1/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.169257/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-06-03T05:34:00.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:34:00.288 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:34:00.282+0000 7f8355a777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:34:00.316 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:34:00.310+0000 7f8355a777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:34:00.317 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:34:00.310+0000 7f8355a777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:34:00.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-03T05:34:00.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:34:00.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T05:34:00.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:34:00.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:34:00.680 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:34:00.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:34:00.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:34:00.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:34:00.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:34:00.913 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:34:00.906+0000 7f8355a777c0 -1 Falling back to public interface 2024-06-03T05:34:01.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:34:02.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:34:02.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:34:02.071 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:34:02.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:34:02.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:34:02.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:34:02.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:34:03.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:34:03.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:34:03.479 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:34:03.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:34:03.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:34:03.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:34:03.559 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:34:03.554+0000 7f8355a777c0 -1 osd.1 0 log_to_monitors true 2024-06-03T05:34:03.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:34:04.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:34:04.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:34:04.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:34:04.944 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:34:04.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:34:04.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:34:05.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:34:06.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:34:06.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:34:06.470 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:34:06.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:34:06.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:34:06.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:34:06.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:34:07.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:34:07.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:34:07.982 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:34:07.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:34:07.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:34:07.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:34:08.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:34:09.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:34:09.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:34:09.512 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:34:09.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:34:09.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:34:09.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:34:09.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:34:10.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:34:10.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:34:10.974 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:34:10.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:34:10.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:34:10.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:34:11.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:34:12.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:34:12.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:34:12.511 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:34:12.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:34:12.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:34:12.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:34:12.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:34:13.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:34:13.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:34:13.987 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:34:13.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:34:13.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:34:13.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:34:14.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:34:15.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:34:15.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:34:15.532 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:34:15.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:34:15.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:34:15.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:34:15.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:34:16.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:34:16.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:34:16.964 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:34:16.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:34:16.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:34:16.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:34:17.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:34:18.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:34:18.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:34:18.466 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:34:18.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:34:18.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:34:18.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:34:19.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:34:20.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:34:20.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:34:20.316 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:34:20.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:34:20.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:34:20.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:34:21.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:34:22.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:34:22.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:34:22.679 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:34:22.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:34:22.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:34:22.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:34:23.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:34:24.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:34:24.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:34:24.566 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:34:24.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:34:24.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:34:24.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:34:25.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:34:26.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:34:26.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:34:26.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:34:26.495 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:34:26.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:34:26.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:34:27.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:34:28.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:34:28.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:34:28.128 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:34:28.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:34:28.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:34:28.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:34:29.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:34:30.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:34:30.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:34:30.108 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-06-03T05:34:30.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T05:34:30.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:34:30.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:34:30.612 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1492817221,v1:127.0.0.1:6811/1492817221] [v2:127.0.0.1:6812/1492817221,v1:127.0.0.1:6813/1492817221] exists,up 89a4f496-bfd0-4196-93ee-690da357b620 2024-06-03T05:34:30.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:34:30.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:34:30.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:34:30.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T05:34:30.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:52: TEST_snaptrim_stats: run_osd td/test-snaptrim-stats 2 --osd_pool_default_pg_autoscale_mode=off 2024-06-03T05:34:30.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-06-03T05:34:30.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:34:30.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-03T05:34:30.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:34:30.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/2 2024-06-03T05:34:30.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3e288c27-dfcb-4963-a1c0-21f4dfb9cbc8 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-06-03T05:34:30.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:34:30.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:34:30.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:34:30.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/2' 2024-06-03T05:34:30.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/2/journal' 2024-06-03T05:34:30.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:34:30.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:34:30.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-06-03T05:34:30.616 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:34:30.616 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:34:30.616 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:34:30.616 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:34:30.616 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:34:30.616 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169257 2024-06-03T05:34:30.616 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.169257/$cluster-$name.asok' 2024-06-03T05:34:30.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.169257/$cluster-$name.asok' 2024-06-03T05:34:30.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:34:30.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:34:30.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:34:30.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-06-03T05:34:30.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-06-03T05:34:30.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:34:30.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:34:30.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:34:30.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:34:30.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:34:30.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-06-03T05:34:30.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/2 2024-06-03T05:34:30.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:34:30.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=896a999d-d655-4613-96f6-f05910a2014a 2024-06-03T05:34:30.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 896a999d-d655-4613-96f6-f05910a2014a' 2024-06-03T05:34:30.619 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 896a999d-d655-4613-96f6-f05910a2014a 2024-06-03T05:34:30.619 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:34:30.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDmVV1m1Mr0JRAAmdGQy4jnMbKE1ekHU4mUaA== 2024-06-03T05:34:30.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDmVV1m1Mr0JRAAmdGQy4jnMbKE1ekHU4mUaA=="}' 2024-06-03T05:34:30.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 896a999d-d655-4613-96f6-f05910a2014a -i td/test-snaptrim-stats/2/new.json 2024-06-03T05:34:31.132 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:34:31.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/2/new.json 2024-06-03T05:34:31.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=3e288c27-dfcb-4963-a1c0-21f4dfb9cbc8 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/2 --osd-journal=td/test-snaptrim-stats/2/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.169257/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQDmVV1m1Mr0JRAAmdGQy4jnMbKE1ekHU4mUaA== --osd-uuid 896a999d-d655-4613-96f6-f05910a2014a 2024-06-03T05:34:31.177 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:34:31.169+0000 7feba0f287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:34:31.219 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:34:31.213+0000 7feba0f287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:34:31.220 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:34:31.213+0000 7feba0f287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:34:31.220 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:34:31.213+0000 7feba0f287c0 -1 bluestore(td/test-snaptrim-stats/2/block) _read_bdev_label failed to open td/test-snaptrim-stats/2/block: (2) No such file or directory 2024-06-03T05:34:31.221 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:34:31.213+0000 7feba0f287c0 -1 bluestore(td/test-snaptrim-stats/2/block) _read_bdev_label failed to open td/test-snaptrim-stats/2/block: (2) No such file or directory 2024-06-03T05:34:31.221 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:34:31.213+0000 7feba0f287c0 -1 bluestore(td/test-snaptrim-stats/2/block) _read_bdev_label failed to open td/test-snaptrim-stats/2/block: (2) No such file or directory 2024-06-03T05:34:31.258 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:34:31.253+0000 7feba0f287c0 -1 bluestore(td/test-snaptrim-stats/2) _read_fsid unparsable uuid 2024-06-03T05:34:38.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/2/keyring 2024-06-03T05:34:38.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:34:38.169 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-06-03T05:34:38.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-03T05:34:38.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:34:38.741 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-06-03T05:34:38.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-03T05:34:38.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=3e288c27-dfcb-4963-a1c0-21f4dfb9cbc8 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/2 --osd-journal=td/test-snaptrim-stats/2/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.169257/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-06-03T05:34:38.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:34:38.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:34:38.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:34:38.771 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:34:38.765+0000 7fc8ec9ba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:34:38.805 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:34:38.797+0000 7fc8ec9ba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:34:38.806 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:34:38.801+0000 7fc8ec9ba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:34:39.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-03T05:34:39.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:34:39.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T05:34:39.171 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:34:39.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:34:39.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:34:39.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:34:39.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:34:39.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:34:39.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:34:39.355 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:34:39.349+0000 7fc8ec9ba7c0 -1 Falling back to public interface 2024-06-03T05:34:39.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:34:40.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:34:40.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:34:40.598 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:34:40.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:34:40.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:34:40.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:34:41.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:34:41.995 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:34:41.989+0000 7fc8ec9ba7c0 -1 osd.2 0 log_to_monitors true 2024-06-03T05:34:42.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:34:42.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:34:42.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:34:42.003 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:34:42.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:34:42.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:34:42.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:34:43.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:34:43.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:34:43.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:34:43.406 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:34:43.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:34:43.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:34:43.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:34:44.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:34:44.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:34:44.915 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:34:44.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:34:44.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:34:44.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:34:45.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:34:46.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:34:46.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:34:46.418 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:34:46.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:34:46.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:34:46.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:34:46.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:34:47.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:34:47.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:34:47.995 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:34:47.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:34:47.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:34:47.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:34:48.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:34:49.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:34:49.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:34:49.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:34:49.422 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:34:49.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:34:49.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:34:49.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:34:50.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:34:50.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:34:50.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:34:50.919 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:34:50.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:34:50.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:34:51.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:34:52.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:34:52.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:34:52.405 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:34:52.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:34:52.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:34:52.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:34:52.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:34:53.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:34:53.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:34:53.864 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:34:53.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:34:53.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:34:53.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:34:54.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:34:55.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:34:55.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:34:55.416 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:34:55.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:34:55.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:34:55.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:34:56.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:34:57.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:34:57.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:34:57.413 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:34:57.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:34:57.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:34:57.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:34:58.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:34:59.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:34:59.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:34:59.658 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:34:59.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:34:59.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:34:59.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:35:00.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:35:01.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:35:01.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:35:01.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:35:01.699 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:35:01.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:35:01.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:35:02.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:35:03.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:35:03.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:35:03.668 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:35:03.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:35:03.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:35:03.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:35:04.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:35:05.443 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:35:05.443 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:35:05.444 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:35:05.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:35:05.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:35:05.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:35:06.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:35:07.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:35:07.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:35:07.331 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:35:07.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:35:07.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:35:07.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:35:07.958 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/75948083,v1:127.0.0.1:6819/75948083] [v2:127.0.0.1:6820/75948083,v1:127.0.0.1:6821/75948083] exists,up 896a999d-d655-4613-96f6-f05910a2014a 2024-06-03T05:35:07.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:35:07.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:35:07.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:35:07.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:56: TEST_snaptrim_stats: ceph osd set noscrub 2024-06-03T05:35:08.662 INFO:tasks.workunit.client.0.smithi088.stderr:noscrub is set 2024-06-03T05:35:08.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:57: TEST_snaptrim_stats: ceph osd set nodeep-scrub 2024-06-03T05:35:09.416 INFO:tasks.workunit.client.0.smithi088.stderr:nodeep-scrub is set 2024-06-03T05:35:09.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:60: TEST_snaptrim_stats: create_pool test 8 8 2024-06-03T05:35:09.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 8 8 2024-06-03T05:35:10.005 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test' already exists 2024-06-03T05:35:10.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T05:35:11.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:61: TEST_snaptrim_stats: wait_for_clean 2024-06-03T05:35:11.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T05:35:11.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T05:35:11.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T05:35:11.024 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T05:35:11.025 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:35:11.025 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:35:11.025 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:35:11.026 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:35:11.026 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:35:11.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T05:35:11.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T05:35:11.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T05:35:11.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T05:35:11.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T05:35:11.141 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T05:35:11.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T05:35:11.569 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-06-03T05:35:11.569 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-06-03T05:35:11.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T05:35:11.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:35:11.569 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T05:35:11.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836504 2024-06-03T05:35:11.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836504 2024-06-03T05:35:11.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504' 2024-06-03T05:35:11.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:35:11.798 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T05:35:11.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672976 2024-06-03T05:35:11.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672976 2024-06-03T05:35:11.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-42949672976' 2024-06-03T05:35:11.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:35:11.952 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T05:35:12.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509448 2024-06-03T05:35:12.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509448 2024-06-03T05:35:12.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-42949672976 2-64424509448' 2024-06-03T05:35:12.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:35:12.109 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:35:12.109 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836504 2024-06-03T05:35:12.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T05:35:12.110 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836504 2024-06-03T05:35:12.110 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:35:12.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836504 2024-06-03T05:35:12.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-06-03T05:35:12.111 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836504 2024-06-03T05:35:12.112 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:35:12.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836504 2024-06-03T05:35:12.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:35:12.507 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:35:12.507 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672976 2024-06-03T05:35:12.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T05:35:12.509 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672976 2024-06-03T05:35:12.509 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:35:12.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672976 2024-06-03T05:35:12.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-06-03T05:35:12.510 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672976 2024-06-03T05:35:12.510 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T05:35:12.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672974 -lt 42949672976 2024-06-03T05:35:12.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T05:35:13.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T05:35:13.928 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T05:35:14.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672976 2024-06-03T05:35:14.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:35:14.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509448 2024-06-03T05:35:14.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:35:14.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T05:35:14.352 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509448 2024-06-03T05:35:14.353 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:35:14.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509448 2024-06-03T05:35:14.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-06-03T05:35:14.354 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509448 2024-06-03T05:35:14.354 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T05:35:14.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509449 -lt 64424509448 2024-06-03T05:35:14.758 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T05:35:14.759 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:35:14.759 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:35:15.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-06-03T05:35:15.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T05:35:15.287 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T05:35:15.287 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T05:35:15.287 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T05:35:15.287 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T05:35:15.287 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T05:35:15.287 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T05:35:15.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-06-03T05:35:15.706 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T05:35:15.706 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:35:15.707 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:35:16.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-06-03T05:35:16.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T05:35:16.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T05:35:16.263 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:62: TEST_snaptrim_stats: ceph osd dump 2024-06-03T05:35:16.263 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:62: TEST_snaptrim_stats: grep '^pool.*['\'']test['\'']' 2024-06-03T05:35:16.264 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:62: TEST_snaptrim_stats: awk '{ print $2 }' 2024-06-03T05:35:16.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:62: TEST_snaptrim_stats: poolid=1 2024-06-03T05:35:16.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:65: TEST_snaptrim_stats: TESTDATA=testdata.1 2024-06-03T05:35:16.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:66: TEST_snaptrim_stats: dd if=/dev/urandom of=testdata.1 bs=4096 count=1 2024-06-03T05:35:16.646 INFO:tasks.workunit.client.0.smithi088.stderr:1+0 records in 2024-06-03T05:35:16.646 INFO:tasks.workunit.client.0.smithi088.stderr:1+0 records out 2024-06-03T05:35:16.646 INFO:tasks.workunit.client.0.smithi088.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000129012 s, 31.7 MB/s 2024-06-03T05:35:16.646 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: seq 1 10 2024-06-03T05:35:16.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-06-03T05:35:16.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj1 testdata.1 2024-06-03T05:35:16.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-06-03T05:35:16.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj2 testdata.1 2024-06-03T05:35:16.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-06-03T05:35:16.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj3 testdata.1 2024-06-03T05:35:17.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-06-03T05:35:17.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj4 testdata.1 2024-06-03T05:35:17.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-06-03T05:35:17.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj5 testdata.1 2024-06-03T05:35:17.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-06-03T05:35:17.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj6 testdata.1 2024-06-03T05:35:17.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-06-03T05:35:17.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj7 testdata.1 2024-06-03T05:35:17.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-06-03T05:35:17.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj8 testdata.1 2024-06-03T05:35:18.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-06-03T05:35:18.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj9 testdata.1 2024-06-03T05:35:18.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-06-03T05:35:18.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj10 testdata.1 2024-06-03T05:35:18.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:71: TEST_snaptrim_stats: rm -f testdata.1 2024-06-03T05:35:18.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:74: TEST_snaptrim_stats: SNAP=1 2024-06-03T05:35:18.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:75: TEST_snaptrim_stats: rados -p test mksnap snap1 2024-06-03T05:35:18.686 INFO:tasks.workunit.client.0.smithi088.stdout:created pool test snap snap1 2024-06-03T05:35:18.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:76: TEST_snaptrim_stats: TESTDATA=testdata.2 2024-06-03T05:35:18.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:77: TEST_snaptrim_stats: dd if=/dev/urandom of=testdata.2 bs=4096 count=1 2024-06-03T05:35:18.691 INFO:tasks.workunit.client.0.smithi088.stderr:1+0 records in 2024-06-03T05:35:18.691 INFO:tasks.workunit.client.0.smithi088.stderr:1+0 records out 2024-06-03T05:35:18.691 INFO:tasks.workunit.client.0.smithi088.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000137376 s, 29.8 MB/s 2024-06-03T05:35:18.692 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: seq 1 10 2024-06-03T05:35:18.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-06-03T05:35:18.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj1 testdata.2 2024-06-03T05:35:18.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-06-03T05:35:18.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj2 testdata.2 2024-06-03T05:35:19.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-06-03T05:35:19.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj3 testdata.2 2024-06-03T05:35:19.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-06-03T05:35:19.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj4 testdata.2 2024-06-03T05:35:19.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-06-03T05:35:19.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj5 testdata.2 2024-06-03T05:35:19.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-06-03T05:35:19.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj6 testdata.2 2024-06-03T05:35:19.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-06-03T05:35:19.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj7 testdata.2 2024-06-03T05:35:20.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-06-03T05:35:20.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj8 testdata.2 2024-06-03T05:35:20.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-06-03T05:35:20.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj9 testdata.2 2024-06-03T05:35:20.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-06-03T05:35:20.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj10 testdata.2 2024-06-03T05:35:20.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:82: TEST_snaptrim_stats: rm -f testdata.2 2024-06-03T05:35:20.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:85: TEST_snaptrim_stats: rados -p test rmsnap snap1 2024-06-03T05:35:20.725 INFO:tasks.workunit.client.0.smithi088.stdout:removed pool test snap snap1 2024-06-03T05:35:20.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:88: TEST_snaptrim_stats: wait_for_clean 2024-06-03T05:35:20.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T05:35:20.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T05:35:20.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T05:35:20.731 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T05:35:20.731 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:35:20.731 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:35:20.732 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:35:20.732 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:35:20.732 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:35:20.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T05:35:20.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T05:35:20.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T05:35:20.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T05:35:20.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T05:35:20.845 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T05:35:21.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T05:35:21.322 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-06-03T05:35:21.322 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-06-03T05:35:21.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T05:35:21.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:35:21.323 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T05:35:21.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836508 2024-06-03T05:35:21.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836508 2024-06-03T05:35:21.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508' 2024-06-03T05:35:21.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:35:21.542 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T05:35:21.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672980 2024-06-03T05:35:21.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672980 2024-06-03T05:35:21.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672980' 2024-06-03T05:35:21.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:35:21.695 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T05:35:21.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509452 2024-06-03T05:35:21.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509452 2024-06-03T05:35:21.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672980 2-64424509452' 2024-06-03T05:35:21.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:35:21.859 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836508 2024-06-03T05:35:21.859 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:35:21.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T05:35:21.861 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836508 2024-06-03T05:35:21.861 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:35:21.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836508 2024-06-03T05:35:21.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-06-03T05:35:21.862 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836508 2024-06-03T05:35:21.862 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:35:22.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836508 2024-06-03T05:35:22.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:35:22.369 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672980 2024-06-03T05:35:22.369 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:35:22.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T05:35:22.371 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672980 2024-06-03T05:35:22.372 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:35:22.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672980 2024-06-03T05:35:22.373 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672980 2024-06-03T05:35:22.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-06-03T05:35:22.373 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T05:35:22.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672980 -lt 42949672980 2024-06-03T05:35:22.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:35:22.754 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509452 2024-06-03T05:35:22.754 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:35:22.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T05:35:22.755 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509452 2024-06-03T05:35:22.755 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:35:22.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509452 2024-06-03T05:35:22.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-06-03T05:35:22.757 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509452 2024-06-03T05:35:22.758 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T05:35:23.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509452 -lt 64424509452 2024-06-03T05:35:23.195 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T05:35:23.195 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:35:23.196 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:35:23.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-06-03T05:35:23.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T05:35:23.716 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T05:35:23.716 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T05:35:23.716 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T05:35:23.716 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T05:35:23.717 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T05:35:23.717 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T05:35:24.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-06-03T05:35:24.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T05:35:24.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:35:24.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:35:24.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-06-03T05:35:24.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T05:35:24.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T05:35:24.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:89: TEST_snaptrim_stats: sleep 10 2024-06-03T05:35:34.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:90: TEST_snaptrim_stats: local objects_trimmed=0 2024-06-03T05:35:34.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:91: TEST_snaptrim_stats: local snaptrim_duration_total=0.0 2024-06-03T05:35:34.709 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: expr 8 - 1 2024-06-03T05:35:34.711 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: seq 0 7 2024-06-03T05:35:34.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-03T05:35:34.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.0 2024-06-03T05:35:34.714 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.0 query 2024-06-03T05:35:34.714 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-06-03T05:35:34.879 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 0 + 2 2024-06-03T05:35:34.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=2 2024-06-03T05:35:34.881 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-06-03T05:35:34.882 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.0 query 2024-06-03T05:35:34.882 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-06-03T05:35:35.037 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 0.0 + 1.064266891 2024-06-03T05:35:35.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=1.064266891 2024-06-03T05:35:35.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-03T05:35:35.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.1 2024-06-03T05:35:35.039 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.1 query 2024-06-03T05:35:35.039 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-06-03T05:35:35.185 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 2 + 2 2024-06-03T05:35:35.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=4 2024-06-03T05:35:35.187 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-06-03T05:35:35.187 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.1 query 2024-06-03T05:35:35.187 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-06-03T05:35:35.331 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 1.064266891 + 0.408642888 2024-06-03T05:35:35.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=1.472909779 2024-06-03T05:35:35.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-03T05:35:35.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.2 2024-06-03T05:35:35.333 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.2 query 2024-06-03T05:35:35.333 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-06-03T05:35:35.484 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 4 + 0 2024-06-03T05:35:35.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=4 2024-06-03T05:35:35.485 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-06-03T05:35:35.486 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.2 query 2024-06-03T05:35:35.486 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-06-03T05:35:35.637 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 1.472909779 + 0.000592313 2024-06-03T05:35:35.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=1.473502092 2024-06-03T05:35:35.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-03T05:35:35.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.3 2024-06-03T05:35:35.639 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.3 query 2024-06-03T05:35:35.639 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-06-03T05:35:35.789 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 4 + 2 2024-06-03T05:35:35.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=6 2024-06-03T05:35:35.791 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-06-03T05:35:35.792 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.3 query 2024-06-03T05:35:35.792 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-06-03T05:35:35.937 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 1.473502092 + 0.640096945 2024-06-03T05:35:35.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=2.113599037 2024-06-03T05:35:35.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-03T05:35:35.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.4 2024-06-03T05:35:35.938 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-06-03T05:35:35.938 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.4 query 2024-06-03T05:35:36.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 6 + 1 2024-06-03T05:35:36.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=7 2024-06-03T05:35:36.093 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-06-03T05:35:36.094 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.4 query 2024-06-03T05:35:36.094 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-06-03T05:35:36.248 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 2.113599037 + 0.369242883 2024-06-03T05:35:36.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=2.482841920 2024-06-03T05:35:36.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-03T05:35:36.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.5 2024-06-03T05:35:36.250 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.5 query 2024-06-03T05:35:36.250 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-06-03T05:35:36.392 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 7 + 1 2024-06-03T05:35:36.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=8 2024-06-03T05:35:36.394 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-06-03T05:35:36.394 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.5 query 2024-06-03T05:35:36.395 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-06-03T05:35:36.553 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 2.482841920 + 0.244150862 2024-06-03T05:35:36.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=2.726992782 2024-06-03T05:35:36.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-03T05:35:36.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.6 2024-06-03T05:35:36.555 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.6 query 2024-06-03T05:35:36.555 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-06-03T05:35:36.708 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 8 + 0 2024-06-03T05:35:36.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=8 2024-06-03T05:35:36.710 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-06-03T05:35:36.710 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.6 query 2024-06-03T05:35:36.710 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-06-03T05:35:36.860 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 2.726992782 + 0.881480706 2024-06-03T05:35:36.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=3.608473488 2024-06-03T05:35:36.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-03T05:35:36.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.7 2024-06-03T05:35:36.862 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.7 query 2024-06-03T05:35:36.862 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-06-03T05:35:37.014 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 8 + 2 2024-06-03T05:35:37.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=10 2024-06-03T05:35:37.015 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-06-03T05:35:37.016 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.7 query 2024-06-03T05:35:37.016 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-06-03T05:35:37.164 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 3.608473488 + 0.881622502 2024-06-03T05:35:37.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=4.490095990 2024-06-03T05:35:37.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:100: TEST_snaptrim_stats: test 10 -eq 10 2024-06-03T05:35:37.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:101: TEST_snaptrim_stats: echo '4.490095990 > 0.0' 2024-06-03T05:35:37.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:101: TEST_snaptrim_stats: bc 2024-06-03T05:35:37.165 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:35:37.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:103: TEST_snaptrim_stats: teardown td/test-snaptrim-stats 2024-06-03T05:35:37.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-06-03T05:35:37.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:35:37.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-06-03T05:35:37.167 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:35:37.167 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:35:37.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:35:37.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:35:37.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:35:37.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:35:37.478 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:35:37.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:35:37.479 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:35:37.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:35:37.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:35:37.481 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:35:37.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:35:37.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:35:37.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:35:37.483 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:35:37.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:35:37.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:35:37.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-06-03T05:35:37.527 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:35:37.527 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:35:37.527 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169257 2024-06-03T05:35:37.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.169257 2024-06-03T05:35:37.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:35:37.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:35:37.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:34: run: teardown td/test-snaptrim-stats 2024-06-03T05:35:37.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-06-03T05:35:37.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:35:37.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-06-03T05:35:37.530 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:35:37.530 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:35:37.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:35:37.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:35:37.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:35:37.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:35:37.532 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:35:37.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:35:37.533 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:35:37.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:35:37.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:35:37.535 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:35:37.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:35:37.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:35:37.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:35:37.537 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:35:37.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:35:37.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:35:37.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-06-03T05:35:37.540 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:35:37.540 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:35:37.540 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169257 2024-06-03T05:35:37.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.169257 2024-06-03T05:35:37.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:35:37.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:35:37.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:31: run: for func in $funcs 2024-06-03T05:35:37.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:32: run: setup td/test-snaptrim-stats 2024-06-03T05:35:37.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-snaptrim-stats 2024-06-03T05:35:37.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-snaptrim-stats 2024-06-03T05:35:37.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-06-03T05:35:37.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:35:37.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-06-03T05:35:37.543 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:35:37.543 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:35:37.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:35:37.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:35:37.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:35:37.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:35:37.545 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:35:37.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:35:37.546 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:35:37.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:35:37.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:35:37.547 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:35:37.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:35:37.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:35:37.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:35:37.549 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:35:37.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:35:37.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:35:37.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-06-03T05:35:37.553 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:35:37.553 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:35:37.553 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169257 2024-06-03T05:35:37.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.169257 2024-06-03T05:35:37.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:35:37.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:35:37.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-snaptrim-stats 2024-06-03T05:35:37.555 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:35:37.555 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:35:37.556 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169257 2024-06-03T05:35:37.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.169257 2024-06-03T05:35:37.559 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:35:37.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:35:37.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:35:37.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-snaptrim-stats 1' TERM HUP INT 2024-06-03T05:35:37.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:33: run: TEST_snaptrim_stats_multiple_snaps td/test-snaptrim-stats 2024-06-03T05:35:37.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:107: TEST_snaptrim_stats_multiple_snaps: local dir=td/test-snaptrim-stats 2024-06-03T05:35:37.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:108: TEST_snaptrim_stats_multiple_snaps: local poolname=test 2024-06-03T05:35:37.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:109: TEST_snaptrim_stats_multiple_snaps: local OSDS=3 2024-06-03T05:35:37.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:110: TEST_snaptrim_stats_multiple_snaps: local PGNUM=8 2024-06-03T05:35:37.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:111: TEST_snaptrim_stats_multiple_snaps: local PGPNUM=8 2024-06-03T05:35:37.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:112: TEST_snaptrim_stats_multiple_snaps: local objects=10 2024-06-03T05:35:37.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:113: TEST_snaptrim_stats_multiple_snaps: local WAIT_FOR_UPDATE=10 2024-06-03T05:35:37.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:115: TEST_snaptrim_stats_multiple_snaps: setup td/test-snaptrim-stats 2024-06-03T05:35:37.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-snaptrim-stats 2024-06-03T05:35:37.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-snaptrim-stats 2024-06-03T05:35:37.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-06-03T05:35:37.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:35:37.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-06-03T05:35:37.561 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:35:37.561 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:35:37.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:35:37.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:35:37.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:35:37.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:35:37.562 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:35:37.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:35:37.563 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:35:37.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:35:37.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:35:37.564 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:35:37.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:35:37.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:35:37.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:35:37.566 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:35:37.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:35:37.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:35:37.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-06-03T05:35:37.569 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:35:37.569 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:35:37.569 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169257 2024-06-03T05:35:37.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.169257 2024-06-03T05:35:37.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:35:37.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:35:37.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-snaptrim-stats 2024-06-03T05:35:37.572 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:35:37.573 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:35:37.573 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169257 2024-06-03T05:35:37.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.169257 2024-06-03T05:35:37.574 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:35:37.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:35:37.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:35:37.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-snaptrim-stats 1' TERM HUP INT 2024-06-03T05:35:37.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:116: TEST_snaptrim_stats_multiple_snaps: run_mon td/test-snaptrim-stats a --osd_pool_default_size=3 2024-06-03T05:35:37.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-snaptrim-stats 2024-06-03T05:35:37.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:35:37.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:35:37.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:35:37.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-snaptrim-stats/a 2024-06-03T05:35:37.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-snaptrim-stats/a --run-dir=td/test-snaptrim-stats --osd_pool_default_size=3 2024-06-03T05:35:38.011 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:35:38.011 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:35:38.012 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:35:38.012 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:35:38.012 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:35:38.012 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169257 2024-06-03T05:35:38.012 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.169257/$cluster-$name.asok' 2024-06-03T05:35:38.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-snaptrim-stats/a '--log-file=td/test-snaptrim-stats/$name.log' '--admin-socket=/tmp/ceph-asok.169257/$cluster-$name.asok' --mon-cluster-log-file=td/test-snaptrim-stats/log --run-dir=td/test-snaptrim-stats '--pid-file=td/test-snaptrim-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-06-03T05:35:38.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:35:38.213 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:35:38.214 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:35:38.214 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:35:38.214 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:35:38.215 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:35:38.215 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:35:38.215 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:35:38.215 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:35:38.215 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:35:38.215 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:35:38.215 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169257 2024-06-03T05:35:38.215 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.169257/ceph-mon.a.asok 2024-06-03T05:35:38.216 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:35:38.216 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.169257/ceph-mon.a.asok config get fsid 2024-06-03T05:35:38.333 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:35:38.333 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:35:38.333 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:35:38.333 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:35:38.334 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:35:38.334 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:35:38.334 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:35:38.334 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:35:38.334 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:35:38.334 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:35:38.334 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169257 2024-06-03T05:35:38.335 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.169257/ceph-mon.a.asok 2024-06-03T05:35:38.335 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:35:38.335 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.169257/ceph-mon.a.asok config get mon_host 2024-06-03T05:35:38.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:117: TEST_snaptrim_stats_multiple_snaps: run_mgr td/test-snaptrim-stats x 2024-06-03T05:35:38.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-snaptrim-stats 2024-06-03T05:35:38.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T05:35:38.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T05:35:38.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T05:35:38.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-snaptrim-stats/x 2024-06-03T05:35:38.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T05:35:39.371 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T05:35:39.371 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:35:39.371 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:35:39.371 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:35:39.371 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:35:39.372 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169257 2024-06-03T05:35:39.372 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.169257/$cluster-$name.asok' 2024-06-03T05:35:39.372 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:35:39.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-snaptrim-stats/x '--log-file=td/test-snaptrim-stats/$name.log' '--admin-socket=/tmp/ceph-asok.169257/$cluster-$name.asok' --run-dir=td/test-snaptrim-stats '--pid-file=td/test-snaptrim-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:35:39.404 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: expr 3 - 1 2024-06-03T05:35:39.405 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: seq 0 2 2024-06-03T05:35:39.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T05:35:39.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:120: TEST_snaptrim_stats_multiple_snaps: run_osd td/test-snaptrim-stats 0 --osd_pool_default_pg_autoscale_mode=off 2024-06-03T05:35:39.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-06-03T05:35:39.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:35:39.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T05:35:39.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:35:39.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/0 2024-06-03T05:35:39.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3e288c27-dfcb-4963-a1c0-21f4dfb9cbc8 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-06-03T05:35:39.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:35:39.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:35:39.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:35:39.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/0' 2024-06-03T05:35:39.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/0/journal' 2024-06-03T05:35:39.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:35:39.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:35:39.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-06-03T05:35:39.410 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:35:39.410 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:35:39.410 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:35:39.410 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:35:39.410 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:35:39.410 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169257 2024-06-03T05:35:39.410 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.169257/$cluster-$name.asok' 2024-06-03T05:35:39.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.169257/$cluster-$name.asok' 2024-06-03T05:35:39.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:35:39.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:35:39.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:35:39.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-06-03T05:35:39.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-06-03T05:35:39.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:35:39.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:35:39.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:35:39.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:35:39.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:35:39.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-06-03T05:35:39.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/0 2024-06-03T05:35:39.412 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:35:39.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3babf7a1-19f1-4b05-b283-313f6504bc2c 2024-06-03T05:35:39.412 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 3babf7a1-19f1-4b05-b283-313f6504bc2c 2024-06-03T05:35:39.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3babf7a1-19f1-4b05-b283-313f6504bc2c' 2024-06-03T05:35:39.412 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:35:39.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQArVl1m04hPGRAAxGJEIc0rOq30zAv8R0muaA== 2024-06-03T05:35:39.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQArVl1m04hPGRAAxGJEIc0rOq30zAv8R0muaA=="}' 2024-06-03T05:35:39.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3babf7a1-19f1-4b05-b283-313f6504bc2c -i td/test-snaptrim-stats/0/new.json 2024-06-03T05:35:39.744 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:35:39.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/0/new.json 2024-06-03T05:35:39.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=3e288c27-dfcb-4963-a1c0-21f4dfb9cbc8 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/0 --osd-journal=td/test-snaptrim-stats/0/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.169257/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQArVl1m04hPGRAAxGJEIc0rOq30zAv8R0muaA== --osd-uuid 3babf7a1-19f1-4b05-b283-313f6504bc2c 2024-06-03T05:35:39.794 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:35:39.789+0000 7fd1fe9af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:35:39.821 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:35:39.813+0000 7fd1fe9af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:35:39.822 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:35:39.817+0000 7fd1fe9af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:35:39.822 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:35:39.817+0000 7fd1fe9af7c0 -1 bluestore(td/test-snaptrim-stats/0/block) _read_bdev_label failed to open td/test-snaptrim-stats/0/block: (2) No such file or directory 2024-06-03T05:35:39.822 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:35:39.817+0000 7fd1fe9af7c0 -1 bluestore(td/test-snaptrim-stats/0/block) _read_bdev_label failed to open td/test-snaptrim-stats/0/block: (2) No such file or directory 2024-06-03T05:35:39.823 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:35:39.817+0000 7fd1fe9af7c0 -1 bluestore(td/test-snaptrim-stats/0/block) _read_bdev_label failed to open td/test-snaptrim-stats/0/block: (2) No such file or directory 2024-06-03T05:35:39.862 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:35:39.853+0000 7fd1fe9af7c0 -1 bluestore(td/test-snaptrim-stats/0) _read_fsid unparsable uuid 2024-06-03T05:35:46.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/0/keyring 2024-06-03T05:35:46.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:35:46.191 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T05:35:46.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T05:35:46.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:35:46.730 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:35:46.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T05:35:46.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=3e288c27-dfcb-4963-a1c0-21f4dfb9cbc8 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/0 --osd-journal=td/test-snaptrim-stats/0/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.169257/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-06-03T05:35:46.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:35:46.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:35:46.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:35:46.761 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:35:46.753+0000 7ff17bf7b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:35:46.764 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:35:46.757+0000 7ff17bf7b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:35:46.765 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:35:46.757+0000 7ff17bf7b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:35:47.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T05:35:47.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:35:47.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:35:47.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:35:47.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:35:47.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:35:47.117 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:35:47.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:35:47.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:35:47.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:35:47.305 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:35:47.297+0000 7ff17bf7b7c0 -1 Falling back to public interface 2024-06-03T05:35:47.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:35:48.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:35:48.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:35:48.531 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:35:48.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:35:48.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:35:48.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:35:48.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:35:49.921 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:35:49.913+0000 7ff17bf7b7c0 -1 osd.0 0 log_to_monitors true 2024-06-03T05:35:49.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:35:49.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:35:49.924 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:35:49.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:35:49.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:35:49.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:35:50.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:35:51.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:35:51.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:35:51.427 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:35:51.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:35:51.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:35:51.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:35:51.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:35:52.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:35:52.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:35:52.924 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:35:52.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:35:52.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:35:52.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:35:53.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:35:54.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:35:54.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:35:54.366 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:35:54.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:35:54.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:35:54.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:35:54.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:35:55.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:35:55.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:35:55.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:35:55.835 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:35:55.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:35:55.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:35:56.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:35:57.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:35:57.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:35:57.283 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:35:57.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:35:57.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:35:57.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:35:57.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:35:58.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:35:58.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:35:58.758 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:35:58.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:35:58.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:35:58.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:35:59.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:36:00.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:36:00.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:36:00.212 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:36:00.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:36:00.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:36:00.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:36:00.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:36:01.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:36:01.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:36:01.767 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:36:01.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:36:01.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:36:01.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:36:02.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:36:03.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:36:03.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:36:03.208 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:36:03.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:36:03.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:36:03.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:36:03.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:36:04.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:36:04.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:36:04.663 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:36:04.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:36:04.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:36:04.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:36:05.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:36:06.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:36:06.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:36:06.530 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:36:06.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:36:06.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:36:06.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:36:07.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:36:08.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:36:08.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:36:08.630 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:36:08.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:36:08.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:36:08.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:36:09.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:36:10.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:36:10.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:36:10.066 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:36:10.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:36:10.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:36:10.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:36:10.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:36:11.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:36:11.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:36:11.958 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:36:11.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:36:11.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:36:11.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:36:12.462 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:36:13.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:36:13.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:36:13.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:36:13.464 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:36:13.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:36:13.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:36:14.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:36:15.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:36:15.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:36:15.479 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-06-03T05:36:15.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T05:36:15.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:36:15.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:36:15.972 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/8804095,v1:127.0.0.1:6803/8804095] [v2:127.0.0.1:6804/8804095,v1:127.0.0.1:6805/8804095] exists,up 3babf7a1-19f1-4b05-b283-313f6504bc2c 2024-06-03T05:36:15.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:36:15.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:36:15.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:36:15.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T05:36:15.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:120: TEST_snaptrim_stats_multiple_snaps: run_osd td/test-snaptrim-stats 1 --osd_pool_default_pg_autoscale_mode=off 2024-06-03T05:36:15.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-06-03T05:36:15.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:36:15.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-03T05:36:15.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:36:15.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/1 2024-06-03T05:36:15.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3e288c27-dfcb-4963-a1c0-21f4dfb9cbc8 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-06-03T05:36:15.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:36:15.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:36:15.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:36:15.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/1' 2024-06-03T05:36:15.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/1/journal' 2024-06-03T05:36:15.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:36:15.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:36:15.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-06-03T05:36:15.977 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:36:15.977 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:36:15.977 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:36:15.977 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:36:15.977 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:36:15.977 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169257 2024-06-03T05:36:15.978 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.169257/$cluster-$name.asok' 2024-06-03T05:36:15.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.169257/$cluster-$name.asok' 2024-06-03T05:36:15.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:36:15.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:36:15.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:36:15.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-06-03T05:36:15.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-06-03T05:36:15.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:36:15.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:36:15.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:36:15.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:36:15.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:36:15.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-06-03T05:36:15.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/1 2024-06-03T05:36:15.979 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:36:15.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7220dc8b-e400-4439-88a7-74a390e0e40e 2024-06-03T05:36:15.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 7220dc8b-e400-4439-88a7-74a390e0e40e' 2024-06-03T05:36:15.980 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 7220dc8b-e400-4439-88a7-74a390e0e40e 2024-06-03T05:36:15.980 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:36:15.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBPVl1mlrU1OxAAJ7OcOFdLHdilw6qFJiP2rw== 2024-06-03T05:36:15.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBPVl1mlrU1OxAAJ7OcOFdLHdilw6qFJiP2rw=="}' 2024-06-03T05:36:15.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7220dc8b-e400-4439-88a7-74a390e0e40e -i td/test-snaptrim-stats/1/new.json 2024-06-03T05:36:16.551 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:36:16.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/1/new.json 2024-06-03T05:36:16.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=3e288c27-dfcb-4963-a1c0-21f4dfb9cbc8 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/1 --osd-journal=td/test-snaptrim-stats/1/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.169257/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQBPVl1mlrU1OxAAJ7OcOFdLHdilw6qFJiP2rw== --osd-uuid 7220dc8b-e400-4439-88a7-74a390e0e40e 2024-06-03T05:36:16.593 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:36:16.585+0000 7f7d938627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:36:16.648 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:36:16.641+0000 7f7d938627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:36:16.650 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:36:16.645+0000 7f7d938627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:36:16.651 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:36:16.645+0000 7f7d938627c0 -1 bluestore(td/test-snaptrim-stats/1/block) _read_bdev_label failed to open td/test-snaptrim-stats/1/block: (2) No such file or directory 2024-06-03T05:36:16.651 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:36:16.645+0000 7f7d938627c0 -1 bluestore(td/test-snaptrim-stats/1/block) _read_bdev_label failed to open td/test-snaptrim-stats/1/block: (2) No such file or directory 2024-06-03T05:36:16.651 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:36:16.645+0000 7f7d938627c0 -1 bluestore(td/test-snaptrim-stats/1/block) _read_bdev_label failed to open td/test-snaptrim-stats/1/block: (2) No such file or directory 2024-06-03T05:36:16.695 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:36:16.689+0000 7f7d938627c0 -1 bluestore(td/test-snaptrim-stats/1) _read_fsid unparsable uuid 2024-06-03T05:36:23.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/1/keyring 2024-06-03T05:36:23.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:36:23.469 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-06-03T05:36:23.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-03T05:36:23.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:36:24.045 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-06-03T05:36:24.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-03T05:36:24.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=3e288c27-dfcb-4963-a1c0-21f4dfb9cbc8 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/1 --osd-journal=td/test-snaptrim-stats/1/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.169257/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-06-03T05:36:24.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:36:24.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:36:24.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:36:24.075 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:36:24.069+0000 7f40903977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:36:24.096 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:36:24.089+0000 7f40903977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:36:24.097 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:36:24.089+0000 7f40903977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:36:24.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-03T05:36:24.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:36:24.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T05:36:24.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:36:24.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:36:24.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:36:24.474 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:36:24.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:36:24.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:36:24.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:36:24.656 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:36:24.649+0000 7f40903977c0 -1 Falling back to public interface 2024-06-03T05:36:24.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:36:25.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:36:25.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:36:25.877 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:36:25.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:36:25.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:36:25.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:36:26.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:36:27.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:36:27.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:36:27.281 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:36:27.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:36:27.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:36:27.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:36:27.298 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:36:27.289+0000 7f40903977c0 -1 osd.1 0 log_to_monitors true 2024-06-03T05:36:27.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:36:28.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:36:28.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:36:28.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:36:28.814 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:36:28.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:36:28.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:36:29.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:36:30.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:36:30.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:36:30.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:36:30.349 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:36:30.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:36:30.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:36:30.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:36:31.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:36:31.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:36:31.907 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:36:31.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:36:31.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:36:31.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:36:32.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:36:33.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:36:33.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:36:33.523 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:36:33.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:36:33.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:36:33.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:36:33.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:36:34.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:36:34.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:36:34.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:36:34.945 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:36:34.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:36:34.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:36:35.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:36:36.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:36:36.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:36:36.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:36:36.428 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:36:36.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:36:36.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:36:36.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:36:37.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:36:37.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:36:37.898 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:36:37.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:36:37.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:36:37.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:36:38.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:36:39.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:36:39.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:36:39.552 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:36:39.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:36:39.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:36:39.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:36:40.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:36:41.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:36:41.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:36:41.196 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:36:41.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:36:41.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:36:41.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:36:41.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:36:42.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:36:42.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:36:42.854 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:36:42.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:36:42.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:36:42.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:36:43.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:36:44.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:36:44.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:36:44.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:36:44.710 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:36:44.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:36:44.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:36:45.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:36:46.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:36:46.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:36:46.840 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:36:46.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:36:46.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:36:46.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:36:47.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:36:48.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:36:48.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:36:48.640 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:36:48.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:36:48.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:36:48.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:36:49.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:36:50.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:36:50.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:36:50.631 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:36:50.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:36:50.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:36:50.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:36:51.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:36:52.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:36:52.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:36:52.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:36:52.839 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:36:52.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:36:52.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:36:53.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:36:54.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:36:54.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:36:54.680 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-06-03T05:36:54.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T05:36:54.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:36:54.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:36:55.082 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1819166425,v1:127.0.0.1:6811/1819166425] [v2:127.0.0.1:6812/1819166425,v1:127.0.0.1:6813/1819166425] exists,up 7220dc8b-e400-4439-88a7-74a390e0e40e 2024-06-03T05:36:55.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:36:55.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:36:55.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:36:55.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-03T05:36:55.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:120: TEST_snaptrim_stats_multiple_snaps: run_osd td/test-snaptrim-stats 2 --osd_pool_default_pg_autoscale_mode=off 2024-06-03T05:36:55.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-06-03T05:36:55.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:36:55.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-03T05:36:55.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:36:55.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/2 2024-06-03T05:36:55.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3e288c27-dfcb-4963-a1c0-21f4dfb9cbc8 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-06-03T05:36:55.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:36:55.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:36:55.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:36:55.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/2' 2024-06-03T05:36:55.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/2/journal' 2024-06-03T05:36:55.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:36:55.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:36:55.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-06-03T05:36:55.088 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:36:55.088 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:36:55.088 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:36:55.088 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:36:55.088 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:36:55.088 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169257 2024-06-03T05:36:55.089 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.169257/$cluster-$name.asok' 2024-06-03T05:36:55.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.169257/$cluster-$name.asok' 2024-06-03T05:36:55.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:36:55.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:36:55.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:36:55.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-06-03T05:36:55.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-06-03T05:36:55.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:36:55.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:36:55.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:36:55.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:36:55.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:36:55.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-06-03T05:36:55.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/2 2024-06-03T05:36:55.090 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:36:55.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e9944717-adc8-4943-ad23-ca41334520c9 2024-06-03T05:36:55.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 e9944717-adc8-4943-ad23-ca41334520c9' 2024-06-03T05:36:55.091 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 e9944717-adc8-4943-ad23-ca41334520c9 2024-06-03T05:36:55.091 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:36:55.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB3Vl1mpp0ABhAAv7Qmt8u6lpdV2gwUvdjPsA== 2024-06-03T05:36:55.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB3Vl1mpp0ABhAAv7Qmt8u6lpdV2gwUvdjPsA=="}' 2024-06-03T05:36:55.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e9944717-adc8-4943-ad23-ca41334520c9 -i td/test-snaptrim-stats/2/new.json 2024-06-03T05:36:55.591 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:36:55.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/2/new.json 2024-06-03T05:36:55.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=3e288c27-dfcb-4963-a1c0-21f4dfb9cbc8 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/2 --osd-journal=td/test-snaptrim-stats/2/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.169257/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQB3Vl1mpp0ABhAAv7Qmt8u6lpdV2gwUvdjPsA== --osd-uuid e9944717-adc8-4943-ad23-ca41334520c9 2024-06-03T05:36:55.641 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:36:55.633+0000 7fad16b337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:36:55.654 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:36:55.649+0000 7fad16b337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:36:55.655 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:36:55.649+0000 7fad16b337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:36:55.656 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:36:55.649+0000 7fad16b337c0 -1 bluestore(td/test-snaptrim-stats/2/block) _read_bdev_label failed to open td/test-snaptrim-stats/2/block: (2) No such file or directory 2024-06-03T05:36:55.656 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:36:55.649+0000 7fad16b337c0 -1 bluestore(td/test-snaptrim-stats/2/block) _read_bdev_label failed to open td/test-snaptrim-stats/2/block: (2) No such file or directory 2024-06-03T05:36:55.656 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:36:55.649+0000 7fad16b337c0 -1 bluestore(td/test-snaptrim-stats/2/block) _read_bdev_label failed to open td/test-snaptrim-stats/2/block: (2) No such file or directory 2024-06-03T05:36:55.694 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:36:55.689+0000 7fad16b337c0 -1 bluestore(td/test-snaptrim-stats/2) _read_fsid unparsable uuid 2024-06-03T05:37:02.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/2/keyring 2024-06-03T05:37:02.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:37:02.436 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-06-03T05:37:02.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-03T05:37:02.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:37:03.008 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-06-03T05:37:03.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-03T05:37:03.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=3e288c27-dfcb-4963-a1c0-21f4dfb9cbc8 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/2 --osd-journal=td/test-snaptrim-stats/2/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.169257/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-06-03T05:37:03.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:37:03.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:37:03.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:37:03.038 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:37:03.033+0000 7fb2f31267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:37:03.057 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:37:03.049+0000 7fb2f31267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:37:03.058 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:37:03.053+0000 7fb2f31267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:37:03.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-03T05:37:03.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:37:03.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T05:37:03.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:37:03.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:37:03.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:37:03.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:37:03.445 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:37:03.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:37:03.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:37:03.640 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:37:03.633+0000 7fb2f31267c0 -1 Falling back to public interface 2024-06-03T05:37:03.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:37:04.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:37:04.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:37:04.873 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:37:04.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:37:04.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:37:04.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:37:05.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:37:06.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:37:06.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:37:06.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:37:06.271 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:37:06.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:37:06.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:37:06.303 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:37:06.297+0000 7fb2f31267c0 -1 osd.2 0 log_to_monitors true 2024-06-03T05:37:06.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:37:07.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:37:07.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:37:07.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:37:07.866 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:37:07.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:37:07.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:37:08.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:37:09.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:37:09.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:37:09.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:37:09.500 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:37:09.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:37:09.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:37:09.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:37:10.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:37:10.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:37:10.931 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:37:10.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:37:10.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:37:10.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:37:11.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:37:12.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:37:12.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:37:12.473 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:37:12.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:37:12.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:37:12.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:37:12.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:37:13.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:37:13.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:37:13.941 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:37:13.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:37:13.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:37:13.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:37:14.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:37:15.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:37:15.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:37:15.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:37:15.749 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:37:15.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:37:15.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:37:16.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:37:17.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:37:17.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:37:17.412 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:37:17.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:37:17.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:37:17.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:37:17.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:37:18.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:37:18.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:37:18.826 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:37:18.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:37:18.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:37:18.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:37:19.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:37:20.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:37:20.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:37:20.347 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:37:20.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:37:20.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:37:20.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:37:21.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:37:22.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:37:22.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:37:22.678 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:37:22.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:37:22.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:37:22.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:37:23.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:37:24.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:37:24.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:37:24.530 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:37:24.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:37:24.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:37:24.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:37:25.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:37:26.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:37:26.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:37:26.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:37:26.370 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:37:26.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:37:26.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:37:27.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:37:28.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:37:28.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:37:28.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:37:28.431 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:37:28.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:37:28.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:37:29.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:37:30.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:37:30.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:37:30.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:37:30.304 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:37:30.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:37:30.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:37:31.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:37:32.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:37:32.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:37:32.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:37:32.609 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:37:32.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:37:32.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:37:33.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:37:34.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:37:34.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:37:34.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T05:37:34.057 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-06-03T05:37:34.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:37:34.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:37:34.467 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4165425624,v1:127.0.0.1:6819/4165425624] [v2:127.0.0.1:6820/4165425624,v1:127.0.0.1:6821/4165425624] exists,up e9944717-adc8-4943-ad23-ca41334520c9 2024-06-03T05:37:34.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:37:34.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:37:34.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:37:34.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:124: TEST_snaptrim_stats_multiple_snaps: ceph osd set noscrub 2024-06-03T05:37:35.200 INFO:tasks.workunit.client.0.smithi088.stderr:noscrub is set 2024-06-03T05:37:35.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:125: TEST_snaptrim_stats_multiple_snaps: ceph osd set nodeep-scrub 2024-06-03T05:37:35.979 INFO:tasks.workunit.client.0.smithi088.stderr:nodeep-scrub is set 2024-06-03T05:37:35.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:128: TEST_snaptrim_stats_multiple_snaps: create_pool test 8 8 2024-06-03T05:37:35.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 8 8 2024-06-03T05:37:36.687 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test' already exists 2024-06-03T05:37:36.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-03T05:37:37.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:129: TEST_snaptrim_stats_multiple_snaps: wait_for_clean 2024-06-03T05:37:37.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T05:37:37.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T05:37:37.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T05:37:37.706 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T05:37:37.707 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:37:37.707 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:37:37.707 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:37:37.707 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:37:37.707 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:37:37.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T05:37:37.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T05:37:37.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T05:37:37.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T05:37:37.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T05:37:37.829 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T05:37:38.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T05:37:38.258 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-06-03T05:37:38.258 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-06-03T05:37:38.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T05:37:38.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:37:38.259 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T05:37:38.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836508 2024-06-03T05:37:38.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836508 2024-06-03T05:37:38.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508' 2024-06-03T05:37:38.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:37:38.587 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T05:37:38.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672977 2024-06-03T05:37:38.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672977 2024-06-03T05:37:38.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672977' 2024-06-03T05:37:38.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:37:38.801 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T05:37:38.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509445 2024-06-03T05:37:38.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509445 2024-06-03T05:37:38.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672977 2-64424509445' 2024-06-03T05:37:38.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:37:38.975 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836508 2024-06-03T05:37:38.976 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:37:38.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T05:37:38.977 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836508 2024-06-03T05:37:38.977 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:37:38.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836508 2024-06-03T05:37:38.978 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836508 2024-06-03T05:37:38.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-06-03T05:37:38.979 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:37:39.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836508 2024-06-03T05:37:39.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T05:37:40.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T05:37:40.376 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:37:40.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836508 2024-06-03T05:37:40.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:37:40.782 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672977 2024-06-03T05:37:40.782 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:37:40.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T05:37:40.785 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672977 2024-06-03T05:37:40.785 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:37:40.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672977 2024-06-03T05:37:40.786 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672977 2024-06-03T05:37:40.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-06-03T05:37:40.786 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T05:37:41.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672978 -lt 42949672977 2024-06-03T05:37:41.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:37:41.216 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509445 2024-06-03T05:37:41.217 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:37:41.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T05:37:41.218 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509445 2024-06-03T05:37:41.218 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:37:41.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509445 2024-06-03T05:37:41.219 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509445 2024-06-03T05:37:41.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-06-03T05:37:41.220 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T05:37:41.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509445 -lt 64424509445 2024-06-03T05:37:41.633 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T05:37:41.633 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:37:41.634 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:37:42.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-06-03T05:37:42.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T05:37:42.174 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T05:37:42.174 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T05:37:42.174 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T05:37:42.174 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T05:37:42.175 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T05:37:42.175 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T05:37:42.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-06-03T05:37:42.607 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T05:37:42.607 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:37:42.607 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:37:43.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-06-03T05:37:43.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T05:37:43.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T05:37:43.140 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:130: TEST_snaptrim_stats_multiple_snaps: grep '^pool.*['\'']test['\'']' 2024-06-03T05:37:43.140 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:130: TEST_snaptrim_stats_multiple_snaps: ceph osd dump 2024-06-03T05:37:43.140 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:130: TEST_snaptrim_stats_multiple_snaps: awk '{ print $2 }' 2024-06-03T05:37:43.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:130: TEST_snaptrim_stats_multiple_snaps: poolid=1 2024-06-03T05:37:43.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:133: TEST_snaptrim_stats_multiple_snaps: local TESTDATA=testdata.0 2024-06-03T05:37:43.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:134: TEST_snaptrim_stats_multiple_snaps: dd if=/dev/urandom of=testdata.0 bs=4096 count=1 2024-06-03T05:37:43.600 INFO:tasks.workunit.client.0.smithi088.stderr:1+0 records in 2024-06-03T05:37:43.600 INFO:tasks.workunit.client.0.smithi088.stderr:1+0 records out 2024-06-03T05:37:43.600 INFO:tasks.workunit.client.0.smithi088.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 8.6634e-05 s, 47.3 MB/s 2024-06-03T05:37:43.600 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: seq 1 10 2024-06-03T05:37:43.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-03T05:37:43.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj1 testdata.0 2024-06-03T05:37:43.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-03T05:37:43.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj2 testdata.0 2024-06-03T05:37:43.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-03T05:37:43.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj3 testdata.0 2024-06-03T05:37:44.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-03T05:37:44.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj4 testdata.0 2024-06-03T05:37:44.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-03T05:37:44.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj5 testdata.0 2024-06-03T05:37:44.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-03T05:37:44.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj6 testdata.0 2024-06-03T05:37:44.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-03T05:37:44.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj7 testdata.0 2024-06-03T05:37:44.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-03T05:37:44.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj8 testdata.0 2024-06-03T05:37:45.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-03T05:37:45.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj9 testdata.0 2024-06-03T05:37:45.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-03T05:37:45.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj10 testdata.0 2024-06-03T05:37:45.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:139: TEST_snaptrim_stats_multiple_snaps: rm -f testdata.0 2024-06-03T05:37:45.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:142: TEST_snaptrim_stats_multiple_snaps: NUMSNAPS=2 2024-06-03T05:37:45.397 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:143: TEST_snaptrim_stats_multiple_snaps: seq 1 2 2024-06-03T05:37:45.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:143: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $NUMSNAPS` 2024-06-03T05:37:45.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:145: TEST_snaptrim_stats_multiple_snaps: rados -p test mksnap snap1 2024-06-03T05:37:45.562 INFO:tasks.workunit.client.0.smithi088.stdout:created pool test snap snap1 2024-06-03T05:37:45.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:146: TEST_snaptrim_stats_multiple_snaps: TESTDATA=testdata.1 2024-06-03T05:37:45.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:147: TEST_snaptrim_stats_multiple_snaps: dd if=/dev/urandom of=testdata.1 bs=4096 count=1 2024-06-03T05:37:45.568 INFO:tasks.workunit.client.0.smithi088.stderr:1+0 records in 2024-06-03T05:37:45.569 INFO:tasks.workunit.client.0.smithi088.stderr:1+0 records out 2024-06-03T05:37:45.569 INFO:tasks.workunit.client.0.smithi088.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000237932 s, 17.2 MB/s 2024-06-03T05:37:45.569 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: seq 1 10 2024-06-03T05:37:45.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-03T05:37:45.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj1 testdata.1 2024-06-03T05:37:45.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-03T05:37:45.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj2 testdata.1 2024-06-03T05:37:46.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-03T05:37:46.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj3 testdata.1 2024-06-03T05:37:46.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-03T05:37:46.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj4 testdata.1 2024-06-03T05:37:46.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-03T05:37:46.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj5 testdata.1 2024-06-03T05:37:46.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-03T05:37:46.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj6 testdata.1 2024-06-03T05:37:46.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-03T05:37:46.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj7 testdata.1 2024-06-03T05:37:46.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-03T05:37:46.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj8 testdata.1 2024-06-03T05:37:47.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-03T05:37:47.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj9 testdata.1 2024-06-03T05:37:47.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-03T05:37:47.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj10 testdata.1 2024-06-03T05:37:47.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:152: TEST_snaptrim_stats_multiple_snaps: rm -f testdata.1 2024-06-03T05:37:47.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:143: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $NUMSNAPS` 2024-06-03T05:37:47.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:145: TEST_snaptrim_stats_multiple_snaps: rados -p test mksnap snap2 2024-06-03T05:37:47.733 INFO:tasks.workunit.client.0.smithi088.stdout:created pool test snap snap2 2024-06-03T05:37:47.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:146: TEST_snaptrim_stats_multiple_snaps: TESTDATA=testdata.2 2024-06-03T05:37:47.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:147: TEST_snaptrim_stats_multiple_snaps: dd if=/dev/urandom of=testdata.2 bs=4096 count=1 2024-06-03T05:37:47.738 INFO:tasks.workunit.client.0.smithi088.stderr:1+0 records in 2024-06-03T05:37:47.738 INFO:tasks.workunit.client.0.smithi088.stderr:1+0 records out 2024-06-03T05:37:47.738 INFO:tasks.workunit.client.0.smithi088.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000173532 s, 23.6 MB/s 2024-06-03T05:37:47.739 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: seq 1 10 2024-06-03T05:37:47.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-03T05:37:47.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj1 testdata.2 2024-06-03T05:37:48.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-03T05:37:48.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj2 testdata.2 2024-06-03T05:37:48.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-03T05:37:48.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj3 testdata.2 2024-06-03T05:37:48.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-03T05:37:48.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj4 testdata.2 2024-06-03T05:37:48.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-03T05:37:48.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj5 testdata.2 2024-06-03T05:37:48.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-03T05:37:48.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj6 testdata.2 2024-06-03T05:37:48.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-03T05:37:48.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj7 testdata.2 2024-06-03T05:37:49.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-03T05:37:49.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj8 testdata.2 2024-06-03T05:37:49.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-03T05:37:49.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj9 testdata.2 2024-06-03T05:37:49.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-03T05:37:49.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj10 testdata.2 2024-06-03T05:37:49.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:152: TEST_snaptrim_stats_multiple_snaps: rm -f testdata.2 2024-06-03T05:37:49.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:156: TEST_snaptrim_stats_multiple_snaps: local total_objects_trimmed=0 2024-06-03T05:37:49.622 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:157: TEST_snaptrim_stats_multiple_snaps: seq 1 2 2024-06-03T05:37:49.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:157: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $NUMSNAPS` 2024-06-03T05:37:49.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:159: TEST_snaptrim_stats_multiple_snaps: rados -p test rmsnap snap1 2024-06-03T05:37:49.787 INFO:tasks.workunit.client.0.smithi088.stdout:removed pool test snap snap1 2024-06-03T05:37:49.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:162: TEST_snaptrim_stats_multiple_snaps: wait_for_clean 2024-06-03T05:37:49.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T05:37:49.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T05:37:49.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T05:37:49.791 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T05:37:49.792 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:37:49.792 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:37:49.793 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:37:49.793 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:37:49.793 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:37:49.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T05:37:49.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T05:37:49.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T05:37:49.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T05:37:49.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T05:37:49.913 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T05:37:50.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T05:37:50.335 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-06-03T05:37:50.335 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-06-03T05:37:50.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T05:37:50.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:37:50.336 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T05:37:50.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836513 2024-06-03T05:37:50.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836513 2024-06-03T05:37:50.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513' 2024-06-03T05:37:50.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:37:50.494 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T05:37:50.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672982 2024-06-03T05:37:50.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672982 2024-06-03T05:37:50.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513 1-42949672982' 2024-06-03T05:37:50.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:37:50.647 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T05:37:50.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509449 2024-06-03T05:37:50.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509449 2024-06-03T05:37:50.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513 1-42949672982 2-64424509449' 2024-06-03T05:37:50.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:37:50.943 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836513 2024-06-03T05:37:50.943 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:37:50.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T05:37:50.945 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836513 2024-06-03T05:37:50.945 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:37:50.947 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836513 2024-06-03T05:37:50.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836513 2024-06-03T05:37:50.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-06-03T05:37:50.947 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:37:51.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836510 -lt 21474836513 2024-06-03T05:37:51.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T05:37:52.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T05:37:52.394 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:37:52.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836513 -lt 21474836513 2024-06-03T05:37:52.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:37:52.857 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:37:52.857 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672982 2024-06-03T05:37:52.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T05:37:52.859 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672982 2024-06-03T05:37:52.859 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:37:52.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672982 2024-06-03T05:37:52.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-06-03T05:37:52.861 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672982 2024-06-03T05:37:52.861 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T05:37:53.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672982 -lt 42949672982 2024-06-03T05:37:53.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:37:53.294 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509449 2024-06-03T05:37:53.294 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:37:53.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T05:37:53.296 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509449 2024-06-03T05:37:53.296 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:37:53.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509449 2024-06-03T05:37:53.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-06-03T05:37:53.297 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509449 2024-06-03T05:37:53.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T05:37:53.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509449 -lt 64424509449 2024-06-03T05:37:53.708 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T05:37:53.708 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:37:53.708 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:37:54.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-06-03T05:37:54.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T05:37:54.290 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T05:37:54.290 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T05:37:54.290 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T05:37:54.290 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T05:37:54.290 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T05:37:54.290 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T05:37:54.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-06-03T05:37:54.736 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T05:37:54.737 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:37:54.737 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:37:55.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-06-03T05:37:55.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T05:37:55.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T05:37:55.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:163: TEST_snaptrim_stats_multiple_snaps: sleep 10 2024-06-03T05:38:05.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:164: TEST_snaptrim_stats_multiple_snaps: local objects_trimmed=0 2024-06-03T05:38:05.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:165: TEST_snaptrim_stats_multiple_snaps: local snaptrim_duration_total=0.0 2024-06-03T05:38:05.252 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: expr 8 - 1 2024-06-03T05:38:05.254 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: seq 0 7 2024-06-03T05:38:05.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-03T05:38:05.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.0 2024-06-03T05:38:05.257 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-06-03T05:38:05.257 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.0 query 2024-06-03T05:38:05.421 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 0 + 2 2024-06-03T05:38:05.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=2 2024-06-03T05:38:05.423 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-06-03T05:38:05.423 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.0 query 2024-06-03T05:38:05.423 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-06-03T05:38:05.578 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 0.0 + 1.115694121 2024-06-03T05:38:05.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=1.115694121 2024-06-03T05:38:05.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-03T05:38:05.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.1 2024-06-03T05:38:05.579 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.1 query 2024-06-03T05:38:05.579 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-06-03T05:38:05.736 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 2 + 2 2024-06-03T05:38:05.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=4 2024-06-03T05:38:05.738 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-06-03T05:38:05.738 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.1 query 2024-06-03T05:38:05.738 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-06-03T05:38:05.896 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 1.115694121 + 0.561716344 2024-06-03T05:38:05.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=1.677410465 2024-06-03T05:38:05.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-03T05:38:05.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.2 2024-06-03T05:38:05.897 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.2 query 2024-06-03T05:38:05.897 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-06-03T05:38:06.052 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 4 + 0 2024-06-03T05:38:06.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=4 2024-06-03T05:38:06.053 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-06-03T05:38:06.054 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.2 query 2024-06-03T05:38:06.054 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-06-03T05:38:06.204 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 1.677410465 + 0.000591453 2024-06-03T05:38:06.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=1.678001918 2024-06-03T05:38:06.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-03T05:38:06.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.3 2024-06-03T05:38:06.206 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.3 query 2024-06-03T05:38:06.206 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-06-03T05:38:06.360 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 4 + 2 2024-06-03T05:38:06.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=6 2024-06-03T05:38:06.362 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-06-03T05:38:06.362 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.3 query 2024-06-03T05:38:06.363 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-06-03T05:38:06.505 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 1.678001918 + 0.604160258 2024-06-03T05:38:06.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=2.282162176 2024-06-03T05:38:06.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-03T05:38:06.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.4 2024-06-03T05:38:06.506 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.4 query 2024-06-03T05:38:06.506 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-06-03T05:38:06.660 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 6 + 1 2024-06-03T05:38:06.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=7 2024-06-03T05:38:06.662 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-06-03T05:38:06.662 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.4 query 2024-06-03T05:38:06.662 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-06-03T05:38:06.804 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 2.282162176 + 0.930994807 2024-06-03T05:38:06.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=3.213156983 2024-06-03T05:38:06.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-03T05:38:06.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.5 2024-06-03T05:38:06.805 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.5 query 2024-06-03T05:38:06.805 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-06-03T05:38:06.954 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 7 + 1 2024-06-03T05:38:06.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=8 2024-06-03T05:38:06.955 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-06-03T05:38:06.956 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-06-03T05:38:06.956 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.5 query 2024-06-03T05:38:07.105 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 3.213156983 + 0.821996717 2024-06-03T05:38:07.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=4.035153700 2024-06-03T05:38:07.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-03T05:38:07.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.6 2024-06-03T05:38:07.106 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.6 query 2024-06-03T05:38:07.107 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-06-03T05:38:07.249 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 8 + 0 2024-06-03T05:38:07.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=8 2024-06-03T05:38:07.251 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-06-03T05:38:07.251 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.6 query 2024-06-03T05:38:07.251 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-06-03T05:38:07.392 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 4.035153700 + 0.604386912 2024-06-03T05:38:07.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=4.639540612 2024-06-03T05:38:07.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-03T05:38:07.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.7 2024-06-03T05:38:07.393 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.7 query 2024-06-03T05:38:07.393 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-06-03T05:38:07.533 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 8 + 2 2024-06-03T05:38:07.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=10 2024-06-03T05:38:07.535 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-06-03T05:38:07.535 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.7 query 2024-06-03T05:38:07.535 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-06-03T05:38:07.680 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 4.639540612 + 0.60417819 2024-06-03T05:38:07.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=5.243718802 2024-06-03T05:38:07.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:174: TEST_snaptrim_stats_multiple_snaps: test 10 -eq 10 2024-06-03T05:38:07.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:175: TEST_snaptrim_stats_multiple_snaps: echo '5.243718802 > 0.0' 2024-06-03T05:38:07.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:175: TEST_snaptrim_stats_multiple_snaps: bc 2024-06-03T05:38:07.682 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:38:07.682 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:176: TEST_snaptrim_stats_multiple_snaps: expr 0 + 10 2024-06-03T05:38:07.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:176: TEST_snaptrim_stats_multiple_snaps: total_objects_trimmed=10 2024-06-03T05:38:07.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:157: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $NUMSNAPS` 2024-06-03T05:38:07.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:159: TEST_snaptrim_stats_multiple_snaps: rados -p test rmsnap snap2 2024-06-03T05:38:07.862 INFO:tasks.workunit.client.0.smithi088.stdout:removed pool test snap snap2 2024-06-03T05:38:07.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:162: TEST_snaptrim_stats_multiple_snaps: wait_for_clean 2024-06-03T05:38:07.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-03T05:38:07.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-03T05:38:07.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-03T05:38:07.866 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-03T05:38:07.866 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-03T05:38:07.866 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-03T05:38:07.867 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-03T05:38:07.867 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-03T05:38:07.867 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-03T05:38:07.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-03T05:38:07.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-03T05:38:07.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-03T05:38:07.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-03T05:38:07.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-03T05:38:07.980 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-03T05:38:08.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-03T05:38:08.486 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-06-03T05:38:08.486 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-06-03T05:38:08.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-03T05:38:08.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:38:08.487 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-03T05:38:08.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836518 2024-06-03T05:38:08.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836518 2024-06-03T05:38:08.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836518' 2024-06-03T05:38:08.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:38:08.762 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-03T05:38:09.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672987 2024-06-03T05:38:09.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672987 2024-06-03T05:38:09.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836518 1-42949672987' 2024-06-03T05:38:09.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-03T05:38:09.139 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-03T05:38:09.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509455 2024-06-03T05:38:09.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509455 2024-06-03T05:38:09.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836518 1-42949672987 2-64424509455' 2024-06-03T05:38:09.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:38:09.300 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836518 2024-06-03T05:38:09.300 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:38:09.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-03T05:38:09.301 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836518 2024-06-03T05:38:09.301 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:38:09.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836518 2024-06-03T05:38:09.302 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836518 2024-06-03T05:38:09.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836518' 2024-06-03T05:38:09.303 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:38:09.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836516 -lt 21474836518 2024-06-03T05:38:09.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-03T05:38:10.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-03T05:38:10.705 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-03T05:38:11.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836518 -lt 21474836518 2024-06-03T05:38:11.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:38:11.110 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:38:11.110 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672987 2024-06-03T05:38:11.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-03T05:38:11.112 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672987 2024-06-03T05:38:11.112 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:38:11.113 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672987 2024-06-03T05:38:11.113 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672987 2024-06-03T05:38:11.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672987' 2024-06-03T05:38:11.114 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-03T05:38:11.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672988 -lt 42949672987 2024-06-03T05:38:11.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-03T05:38:11.534 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-03T05:38:11.534 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509455 2024-06-03T05:38:11.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-03T05:38:11.536 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509455 2024-06-03T05:38:11.536 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-03T05:38:11.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509455 2024-06-03T05:38:11.537 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509455 2024-06-03T05:38:11.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-06-03T05:38:11.537 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-03T05:38:11.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509455 -lt 64424509455 2024-06-03T05:38:11.956 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-03T05:38:11.956 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:38:11.956 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:38:12.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-06-03T05:38:12.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-03T05:38:12.492 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-03T05:38:12.493 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-03T05:38:12.493 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-03T05:38:12.493 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-03T05:38:12.493 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-03T05:38:12.493 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-03T05:38:12.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-06-03T05:38:12.949 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-03T05:38:12.949 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-03T05:38:12.949 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-03T05:38:13.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-06-03T05:38:13.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-03T05:38:13.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-03T05:38:13.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:163: TEST_snaptrim_stats_multiple_snaps: sleep 10 2024-06-03T05:38:23.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:164: TEST_snaptrim_stats_multiple_snaps: local objects_trimmed=0 2024-06-03T05:38:23.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:165: TEST_snaptrim_stats_multiple_snaps: local snaptrim_duration_total=0.0 2024-06-03T05:38:23.473 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: expr 8 - 1 2024-06-03T05:38:23.474 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: seq 0 7 2024-06-03T05:38:23.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-03T05:38:23.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.0 2024-06-03T05:38:23.477 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.0 query 2024-06-03T05:38:23.477 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-06-03T05:38:23.649 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 0 + 2 2024-06-03T05:38:23.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=2 2024-06-03T05:38:23.650 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-06-03T05:38:23.651 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.0 query 2024-06-03T05:38:23.651 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-06-03T05:38:23.806 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 0.0 + 0.774982772 2024-06-03T05:38:23.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.774982772 2024-06-03T05:38:23.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-03T05:38:23.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.1 2024-06-03T05:38:23.808 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.1 query 2024-06-03T05:38:23.808 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-06-03T05:38:23.966 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 2 + 2 2024-06-03T05:38:23.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=4 2024-06-03T05:38:23.967 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-06-03T05:38:23.967 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.1 query 2024-06-03T05:38:23.967 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-06-03T05:38:24.119 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .774982772 + 0.587425729 2024-06-03T05:38:24.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=1.362408501 2024-06-03T05:38:24.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-03T05:38:24.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.2 2024-06-03T05:38:24.120 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.2 query 2024-06-03T05:38:24.120 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-06-03T05:38:24.270 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 4 + 0 2024-06-03T05:38:24.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=4 2024-06-03T05:38:24.271 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-06-03T05:38:24.272 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.2 query 2024-06-03T05:38:24.272 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-06-03T05:38:24.425 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 1.362408501 + 0.143034871 2024-06-03T05:38:24.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=1.505443372 2024-06-03T05:38:24.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-03T05:38:24.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.3 2024-06-03T05:38:24.427 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.3 query 2024-06-03T05:38:24.427 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-06-03T05:38:24.580 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 4 + 2 2024-06-03T05:38:24.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=6 2024-06-03T05:38:24.581 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-06-03T05:38:24.582 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.3 query 2024-06-03T05:38:24.582 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-06-03T05:38:24.725 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 1.505443372 + 0.895901654 2024-06-03T05:38:24.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=2.401345026 2024-06-03T05:38:24.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-03T05:38:24.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.4 2024-06-03T05:38:24.727 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.4 query 2024-06-03T05:38:24.727 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-06-03T05:38:24.876 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 6 + 1 2024-06-03T05:38:24.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=7 2024-06-03T05:38:24.877 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-06-03T05:38:24.878 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.4 query 2024-06-03T05:38:24.878 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-06-03T05:38:25.027 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 2.401345026 + 0.476237914 2024-06-03T05:38:25.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=2.877582940 2024-06-03T05:38:25.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-03T05:38:25.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.5 2024-06-03T05:38:25.027 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.5 query 2024-06-03T05:38:25.027 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-06-03T05:38:25.174 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 7 + 1 2024-06-03T05:38:25.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=8 2024-06-03T05:38:25.176 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-06-03T05:38:25.176 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.5 query 2024-06-03T05:38:25.176 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-06-03T05:38:25.324 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 2.877582940 + 0.587391367 2024-06-03T05:38:25.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=3.464974307 2024-06-03T05:38:25.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-03T05:38:25.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.6 2024-06-03T05:38:25.325 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.6 query 2024-06-03T05:38:25.325 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-06-03T05:38:25.478 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 8 + 0 2024-06-03T05:38:25.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=8 2024-06-03T05:38:25.480 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-06-03T05:38:25.480 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.6 query 2024-06-03T05:38:25.480 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-06-03T05:38:25.629 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 3.464974307 + 1.029186099 2024-06-03T05:38:25.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=4.494160406 2024-06-03T05:38:25.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-03T05:38:25.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.7 2024-06-03T05:38:25.630 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.7 query 2024-06-03T05:38:25.630 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-06-03T05:38:25.777 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 8 + 2 2024-06-03T05:38:25.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=10 2024-06-03T05:38:25.779 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-06-03T05:38:25.779 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.7 query 2024-06-03T05:38:25.779 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-06-03T05:38:25.931 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 4.494160406 + 1.172201335 2024-06-03T05:38:25.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=5.666361741 2024-06-03T05:38:25.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:174: TEST_snaptrim_stats_multiple_snaps: test 10 -eq 10 2024-06-03T05:38:25.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:175: TEST_snaptrim_stats_multiple_snaps: echo '5.666361741 > 0.0' 2024-06-03T05:38:25.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:175: TEST_snaptrim_stats_multiple_snaps: bc 2024-06-03T05:38:25.933 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:38:25.933 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:176: TEST_snaptrim_stats_multiple_snaps: expr 10 + 10 2024-06-03T05:38:25.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:176: TEST_snaptrim_stats_multiple_snaps: total_objects_trimmed=20 2024-06-03T05:38:25.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:179: TEST_snaptrim_stats_multiple_snaps: test 20 -eq 20 2024-06-03T05:38:25.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:181: TEST_snaptrim_stats_multiple_snaps: teardown td/test-snaptrim-stats 2024-06-03T05:38:25.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-06-03T05:38:25.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:38:25.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-06-03T05:38:25.935 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:38:25.935 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:38:25.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:38:25.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:38:25.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:38:26.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:38:26.043 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:38:26.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:38:26.046 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:38:26.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:38:26.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:38:26.046 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:38:26.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:38:26.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:38:26.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:38:26.047 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:38:26.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:38:26.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:38:26.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-06-03T05:38:26.082 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:38:26.082 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:38:26.082 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169257 2024-06-03T05:38:26.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.169257 2024-06-03T05:38:26.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:38:26.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:38:26.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:34: run: teardown td/test-snaptrim-stats 2024-06-03T05:38:26.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-06-03T05:38:26.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:38:26.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-06-03T05:38:26.084 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:38:26.084 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:38:26.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:38:26.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:38:26.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:38:26.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:38:26.085 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:38:26.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:38:26.086 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:38:26.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:38:26.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:38:26.088 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:38:26.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:38:26.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:38:26.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:38:26.089 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:38:26.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:38:26.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:38:26.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-06-03T05:38:26.091 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:38:26.091 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:38:26.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169257 2024-06-03T05:38:26.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.169257 2024-06-03T05:38:26.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:38:26.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:38:26.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-06-03T05:38:26.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/test-snaptrim-stats 0 2024-06-03T05:38:26.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-06-03T05:38:26.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-06-03T05:38:26.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-06-03T05:38:26.094 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:38:26.094 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:38:26.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:38:26.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:38:26.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:38:26.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:38:26.095 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:38:26.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:38:26.095 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:38:26.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:38:26.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:38:26.097 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:38:26.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:38:26.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:38:26.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:38:26.098 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:38:26.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:38:26.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-06-03T05:38:26.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-06-03T05:38:26.100 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:38:26.100 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:38:26.101 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169257 2024-06-03T05:38:26.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.169257 2024-06-03T05:38:26.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:38:26.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:38:26.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-06-03T05:38:26.102 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-06-03T05:38:26.103 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-06-03T05:38:26.160 INFO:tasks.workunit:Running workunit misc/ver-health.sh... 2024-06-03T05:38:26.160 DEBUG:teuthology.orchestra.run.smithi088:workunit test misc/ver-health.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dfcc4b3c2c55532c4c04fa47551c9df1dffdf746 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh 2024-06-03T05:38:26.213 INFO:tasks.workunit.client.0.smithi088.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-06-03T05:38:26.219 INFO:tasks.workunit.client.0.smithi088.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-06-03T05:38:26.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-03T05:38:26.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-03T05:38:26.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-06-03T05:38:26.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-06-03T05:38:26.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-06-03T05:38:26.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-06-03T05:38:26.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-06-03T05:38:26.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-06-03T05:38:26.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/ver-health 2024-06-03T05:38:26.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:20: run: local dir=td/ver-health 2024-06-03T05:38:26.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:21: run: shift 2024-06-03T05:38:26.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:23: run: export CEPH_MON_A=127.0.0.1:7165 2024-06-03T05:38:26.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:23: run: CEPH_MON_A=127.0.0.1:7165 2024-06-03T05:38:26.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:24: run: export CEPH_MON_B=127.0.0.1:7166 2024-06-03T05:38:26.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:24: run: CEPH_MON_B=127.0.0.1:7166 2024-06-03T05:38:26.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:25: run: export CEPH_ARGS 2024-06-03T05:38:26.220 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:26: run: uuidgen 2024-06-03T05:38:26.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:26: run: CEPH_ARGS+='--fsid=6cd9f48e-d1df-426c-8c75-81e8c5105a2f --auth-supported=none ' 2024-06-03T05:38:26.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:27: run: CEPH_ARGS+='--mon-host= ' 2024-06-03T05:38:26.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:28: run: CEPH_ARGS+='--mon_health_to_clog_tick_interval=1.0 ' 2024-06-03T05:38:26.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:29: run: export 'ORIG_CEPH_ARGS=--fsid=6cd9f48e-d1df-426c-8c75-81e8c5105a2f --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 ' 2024-06-03T05:38:26.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:29: run: ORIG_CEPH_ARGS='--fsid=6cd9f48e-d1df-426c-8c75-81e8c5105a2f --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 ' 2024-06-03T05:38:26.221 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:31: run: set 2024-06-03T05:38:26.222 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-06-03T05:38:26.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:31: run: local 'funcs=TEST_check_version_health_1 2024-06-03T05:38:26.222 INFO:tasks.workunit.client.0.smithi088.stderr:TEST_check_version_health_2 2024-06-03T05:38:26.222 INFO:tasks.workunit.client.0.smithi088.stderr:TEST_check_version_health_3' 2024-06-03T05:38:26.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:32: run: for func in $funcs 2024-06-03T05:38:26.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:33: run: setup td/ver-health 2024-06-03T05:38:26.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2024-06-03T05:38:26.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2024-06-03T05:38:26.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-06-03T05:38:26.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:38:26.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-06-03T05:38:26.223 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:38:26.223 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:38:26.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:38:26.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:38:26.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:38:26.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:38:26.224 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:38:26.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:38:26.225 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:38:26.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:38:26.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:38:26.226 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:38:26.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:38:26.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:38:26.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:38:26.227 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:38:26.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:38:26.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:38:26.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-06-03T05:38:26.229 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:38:26.229 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:38:26.230 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184738 2024-06-03T05:38:26.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.184738 2024-06-03T05:38:26.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:38:26.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:38:26.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2024-06-03T05:38:26.231 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:38:26.232 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:38:26.232 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184738 2024-06-03T05:38:26.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.184738 2024-06-03T05:38:26.232 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:38:26.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-06-03T05:38:26.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-06-03T05:38:26.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:38:26.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2024-06-03T05:38:26.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:34: run: TEST_check_version_health_1 td/ver-health 2024-06-03T05:38:26.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:68: TEST_check_version_health_1: local dir=td/ver-health 2024-06-03T05:38:26.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:71: TEST_check_version_health_1: CEPH_ARGS='--fsid=6cd9f48e-d1df-426c-8c75-81e8c5105a2f --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-06-03T05:38:26.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:73: TEST_check_version_health_1: setup td/ver-health 2024-06-03T05:38:26.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2024-06-03T05:38:26.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2024-06-03T05:38:26.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-06-03T05:38:26.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:38:26.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-06-03T05:38:26.235 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:38:26.235 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:38:26.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:38:26.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:38:26.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:38:26.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:38:26.236 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:38:26.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:38:26.237 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:38:26.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:38:26.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:38:26.238 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:38:26.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:38:26.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:38:26.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:38:26.239 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:38:26.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:38:26.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:38:26.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-06-03T05:38:26.242 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:38:26.242 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:38:26.242 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184738 2024-06-03T05:38:26.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.184738 2024-06-03T05:38:26.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:38:26.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:38:26.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2024-06-03T05:38:26.244 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:38:26.244 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:38:26.244 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184738 2024-06-03T05:38:26.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.184738 2024-06-03T05:38:26.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:38:26.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:38:26.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:38:26.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2024-06-03T05:38:26.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:76: TEST_check_version_health_1: run_mon td/ver-health a --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-06-03T05:38:26.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-06-03T05:38:26.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:38:26.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:38:26.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:38:26.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/a 2024-06-03T05:38:26.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ver-health/a --run-dir=td/ver-health --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-06-03T05:38:26.634 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:38:26.634 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:38:26.634 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:38:26.634 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:38:26.634 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:38:26.634 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184738 2024-06-03T05:38:26.635 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.184738/$cluster-$name.asok' 2024-06-03T05:38:26.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/a '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.184738/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-06-03T05:38:26.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:38:26.864 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:38:26.864 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:38:26.864 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:38:26.864 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:38:26.865 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:38:26.865 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:38:26.866 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:38:26.866 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:38:26.866 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:38:26.866 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:38:26.866 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184738 2024-06-03T05:38:26.866 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.184738/ceph-mon.a.asok 2024-06-03T05:38:26.866 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:38:26.867 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.184738/ceph-mon.a.asok config get fsid 2024-06-03T05:38:26.981 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:38:26.981 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:38:26.981 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:38:26.981 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:38:26.981 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:38:26.982 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:38:26.983 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:38:26.983 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:38:26.983 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:38:26.983 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:38:26.983 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184738 2024-06-03T05:38:26.983 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.184738/ceph-mon.a.asok 2024-06-03T05:38:26.983 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:38:26.983 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.184738/ceph-mon.a.asok config get mon_host 2024-06-03T05:38:27.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:77: TEST_check_version_health_1: run_mon td/ver-health b --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-06-03T05:38:27.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-06-03T05:38:27.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:38:27.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-06-03T05:38:27.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:38:27.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/b 2024-06-03T05:38:27.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ver-health/b --run-dir=td/ver-health --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-06-03T05:38:27.807 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:38:27.807 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:38:27.808 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:38:27.808 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:38:27.808 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:38:27.808 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184738 2024-06-03T05:38:27.808 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.184738/$cluster-$name.asok' 2024-06-03T05:38:27.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/b '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.184738/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-06-03T05:38:28.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:38:28.090 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-06-03T05:38:28.090 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:38:28.090 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-06-03T05:38:28.090 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:38:28.090 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:38:28.090 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-06-03T05:38:28.090 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-06-03T05:38:28.090 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-06-03T05:38:28.091 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:38:28.091 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:38:28.091 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184738 2024-06-03T05:38:28.091 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.184738/ceph-mon.b.asok 2024-06-03T05:38:28.091 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:38:28.091 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.184738/ceph-mon.b.asok config get fsid 2024-06-03T05:38:28.206 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-06-03T05:38:28.206 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:38:28.206 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-06-03T05:38:28.206 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:38:28.206 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:38:28.207 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-06-03T05:38:28.208 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-06-03T05:38:28.208 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-06-03T05:38:28.208 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:38:28.208 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:38:28.208 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184738 2024-06-03T05:38:28.208 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.184738/ceph-mon.b.asok 2024-06-03T05:38:28.208 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:38:28.208 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.184738/ceph-mon.b.asok config get mon_host 2024-06-03T05:38:28.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:78: TEST_check_version_health_1: run_osd td/ver-health 0 2024-06-03T05:38:28.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-06-03T05:38:28.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:38:28.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T05:38:28.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:38:28.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/0 2024-06-03T05:38:28.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6cd9f48e-d1df-426c-8c75-81e8c5105a2f --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-06-03T05:38:28.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:38:28.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:38:28.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:38:28.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/0' 2024-06-03T05:38:28.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2024-06-03T05:38:28.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:38:28.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:38:28.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-06-03T05:38:28.324 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:38:28.324 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:38:28.324 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:38:28.325 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:38:28.325 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:38:28.325 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184738 2024-06-03T05:38:28.325 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.184738/$cluster-$name.asok' 2024-06-03T05:38:28.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.184738/$cluster-$name.asok' 2024-06-03T05:38:28.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:38:28.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:38:28.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:38:28.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-06-03T05:38:28.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-06-03T05:38:28.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:38:28.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:38:28.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:38:28.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:38:28.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:38:28.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:38:28.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/0 2024-06-03T05:38:28.326 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:38:28.326 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 ae186b76-983b-4eb0-a39a-5b34a827afae 2024-06-03T05:38:28.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ae186b76-983b-4eb0-a39a-5b34a827afae 2024-06-03T05:38:28.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ae186b76-983b-4eb0-a39a-5b34a827afae' 2024-06-03T05:38:28.327 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:38:28.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDUVl1mXy9RFBAAspyPjjDXS60mgEeQovl8KQ== 2024-06-03T05:38:28.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDUVl1mXy9RFBAAspyPjjDXS60mgEeQovl8KQ=="}' 2024-06-03T05:38:28.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ae186b76-983b-4eb0-a39a-5b34a827afae -i td/ver-health/0/new.json 2024-06-03T05:38:34.710 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:38:34.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/0/new.json 2024-06-03T05:38:34.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=6cd9f48e-d1df-426c-8c75-81e8c5105a2f --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.184738/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDUVl1mXy9RFBAAspyPjjDXS60mgEeQovl8KQ== --osd-uuid ae186b76-983b-4eb0-a39a-5b34a827afae 2024-06-03T05:38:34.763 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:38:34.757+0000 7f1ab45fa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:38:34.765 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:38:34.757+0000 7f1ab45fa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:38:34.766 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:38:34.761+0000 7f1ab45fa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:38:34.767 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:38:34.761+0000 7f1ab45fa7c0 -1 bluestore(td/ver-health/0/block) _read_bdev_label failed to open td/ver-health/0/block: (2) No such file or directory 2024-06-03T05:38:34.767 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:38:34.761+0000 7f1ab45fa7c0 -1 bluestore(td/ver-health/0/block) _read_bdev_label failed to open td/ver-health/0/block: (2) No such file or directory 2024-06-03T05:38:34.767 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:38:34.761+0000 7f1ab45fa7c0 -1 bluestore(td/ver-health/0/block) _read_bdev_label failed to open td/ver-health/0/block: (2) No such file or directory 2024-06-03T05:38:34.860 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:38:34.857+0000 7f1ab45fa7c0 -1 bluestore(td/ver-health/0) _read_fsid unparsable uuid 2024-06-03T05:38:41.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/0/keyring 2024-06-03T05:38:41.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:38:41.338 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T05:38:41.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T05:38:41.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:38:41.587 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:38:41.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T05:38:41.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:38:41.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=6cd9f48e-d1df-426c-8c75-81e8c5105a2f --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.184738/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:38:41.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:38:41.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:38:41.619 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:38:41.613+0000 7fe5f86227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:38:41.621 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:38:41.613+0000 7fe5f86227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:38:41.623 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:38:41.617+0000 7fe5f86227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:38:41.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T05:38:41.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:38:41.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:38:41.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:38:41.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:38:41.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:38:41.876 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:38:41.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:38:41.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:38:41.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:38:42.176 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:38:42.169+0000 7fe5f86227c0 -1 Falling back to public interface 2024-06-03T05:38:42.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:38:43.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:38:43.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:38:43.179 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:38:43.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:38:43.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:38:43.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:38:43.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:38:44.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:38:44.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:38:44.414 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:38:44.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:38:44.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:38:44.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:38:44.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:38:44.929 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:38:44.921+0000 7fe5f86227c0 -1 osd.0 0 log_to_monitors true 2024-06-03T05:38:45.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:38:45.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:38:45.647 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:38:45.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:38:45.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:38:45.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:38:46.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:38:47.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:38:47.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:38:47.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:38:47.016 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:38:47.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:38:47.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:38:47.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:38:48.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:38:48.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:38:48.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:38:48.248 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:38:48.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:38:48.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:38:48.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:38:49.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:38:49.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:38:49.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:38:49.482 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:38:49.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:38:49.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:38:49.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:38:50.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:38:50.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:38:50.715 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:38:50.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:38:50.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:38:50.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:38:50.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:38:51.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:38:51.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:38:51.949 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:38:51.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:38:51.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:38:51.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:38:52.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:38:53.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:38:53.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:38:53.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:38:53.180 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:38:53.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:38:53.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:38:53.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:38:54.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:38:54.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:38:54.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:38:54.417 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:38:54.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:38:54.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:38:54.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:38:55.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:38:55.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:38:55.650 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:38:55.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:38:55.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:38:55.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:38:55.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:38:56.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:38:56.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:38:56.885 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:38:56.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:38:56.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:38:56.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:38:57.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:38:58.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:38:58.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:38:58.124 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:38:58.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:38:58.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:38:58.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:38:58.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:38:59.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:38:59.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:38:59.360 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:38:59.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:38:59.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:38:59.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:38:59.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:39:00.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:39:00.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:39:00.598 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:39:00.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:39:00.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:39:00.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:39:00.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:39:01.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:39:01.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:39:01.835 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:39:01.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:39:01.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:39:01.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:39:02.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:39:03.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:39:03.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:39:03.072 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:39:03.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:39:03.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:39:03.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:39:03.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:39:04.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:39:04.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:39:04.307 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-06-03T05:39:04.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T05:39:04.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:39:04.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:39:04.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:39:05.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:39:05.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:39:05.571 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-06-03T05:39:05.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-03T05:39:05.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:39:05.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:39:05.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:39:06.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:39:06.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:39:06.806 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-06-03T05:39:06.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-03T05:39:06.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:39:06.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:39:07.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:39:08.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:39:08.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:39:08.039 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-06-03T05:39:08.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-03T05:39:08.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:39:08.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:39:08.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:39:09.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:39:09.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:39:09.280 INFO:tasks.workunit.client.0.smithi088.stdout:22 2024-06-03T05:39:09.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-03T05:39:09.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:39:09.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:39:09.615 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/4080940625,v1:127.0.0.1:6801/4080940625] [v2:127.0.0.1:6802/4080940625,v1:127.0.0.1:6803/4080940625] exists,up ae186b76-983b-4eb0-a39a-5b34a827afae 2024-06-03T05:39:09.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:39:09.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:39:09.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:39:09.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:79: TEST_check_version_health_1: run_osd td/ver-health 1 2024-06-03T05:39:09.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-06-03T05:39:09.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:39:09.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-03T05:39:09.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:39:09.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/1 2024-06-03T05:39:09.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6cd9f48e-d1df-426c-8c75-81e8c5105a2f --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-06-03T05:39:09.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:39:09.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:39:09.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:39:09.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/1' 2024-06-03T05:39:09.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/1/journal' 2024-06-03T05:39:09.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:39:09.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:39:09.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-06-03T05:39:09.619 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:39:09.619 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:39:09.619 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:39:09.619 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:39:09.619 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:39:09.619 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184738 2024-06-03T05:39:09.619 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.184738/$cluster-$name.asok' 2024-06-03T05:39:09.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.184738/$cluster-$name.asok' 2024-06-03T05:39:09.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:39:09.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:39:09.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:39:09.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-06-03T05:39:09.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-06-03T05:39:09.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:39:09.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:39:09.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:39:09.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:39:09.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:39:09.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:39:09.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/1 2024-06-03T05:39:09.622 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:39:09.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2d18a088-38e7-4ff0-a35f-4edb3fb678c0 2024-06-03T05:39:09.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 2d18a088-38e7-4ff0-a35f-4edb3fb678c0' 2024-06-03T05:39:09.623 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 2d18a088-38e7-4ff0-a35f-4edb3fb678c0 2024-06-03T05:39:09.623 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:39:09.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD9Vl1mqz0fJhAA6La50osoPQ+0DCp5MoaTXA== 2024-06-03T05:39:09.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD9Vl1mqz0fJhAA6La50osoPQ+0DCp5MoaTXA=="}' 2024-06-03T05:39:09.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2d18a088-38e7-4ff0-a35f-4edb3fb678c0 -i td/ver-health/1/new.json 2024-06-03T05:39:10.122 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:39:10.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/1/new.json 2024-06-03T05:39:10.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=6cd9f48e-d1df-426c-8c75-81e8c5105a2f --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.184738/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD9Vl1mqz0fJhAA6La50osoPQ+0DCp5MoaTXA== --osd-uuid 2d18a088-38e7-4ff0-a35f-4edb3fb678c0 2024-06-03T05:39:10.180 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:39:10.173+0000 7f4ed9b717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:39:10.182 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:39:10.177+0000 7f4ed9b717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:39:10.184 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:39:10.177+0000 7f4ed9b717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:39:10.184 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:39:10.177+0000 7f4ed9b717c0 -1 bluestore(td/ver-health/1/block) _read_bdev_label failed to open td/ver-health/1/block: (2) No such file or directory 2024-06-03T05:39:10.184 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:39:10.177+0000 7f4ed9b717c0 -1 bluestore(td/ver-health/1/block) _read_bdev_label failed to open td/ver-health/1/block: (2) No such file or directory 2024-06-03T05:39:10.185 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:39:10.177+0000 7f4ed9b717c0 -1 bluestore(td/ver-health/1/block) _read_bdev_label failed to open td/ver-health/1/block: (2) No such file or directory 2024-06-03T05:39:10.301 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:39:10.297+0000 7f4ed9b717c0 -1 bluestore(td/ver-health/1) _read_fsid unparsable uuid 2024-06-03T05:39:16.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/1/keyring 2024-06-03T05:39:16.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:39:16.860 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-06-03T05:39:16.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-03T05:39:16.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:39:17.107 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-06-03T05:39:17.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-03T05:39:17.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=6cd9f48e-d1df-426c-8c75-81e8c5105a2f --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.184738/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:39:17.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:39:17.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:39:17.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:39:17.139 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:39:17.132+0000 7fe2598187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:39:17.141 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:39:17.132+0000 7fe2598187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:39:17.142 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:39:17.136+0000 7fe2598187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:39:17.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-03T05:39:17.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:39:17.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T05:39:17.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:39:17.406 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:39:17.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:39:17.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:39:17.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:39:17.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:39:17.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:39:17.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:39:17.712 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:39:17.704+0000 7fe2598187c0 -1 Falling back to public interface 2024-06-03T05:39:18.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:39:18.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:39:18.686 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:39:18.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:39:18.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:39:18.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:39:18.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:39:19.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:39:19.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:39:19.922 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:39:19.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:39:19.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:39:19.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:39:20.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:39:20.384 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:39:20.376+0000 7fe2598187c0 -1 osd.1 0 log_to_monitors true 2024-06-03T05:39:21.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:39:21.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:39:21.146 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:39:21.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:39:21.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:39:21.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:39:21.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:39:22.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:39:22.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:39:22.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:39:22.537 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:39:22.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:39:22.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:39:22.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:39:23.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:39:23.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:39:23.771 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:39:23.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:39:23.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:39:23.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:39:24.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:39:25.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:39:25.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:39:25.008 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:39:25.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:39:25.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:39:25.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:39:25.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:39:26.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:39:26.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:39:26.243 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:39:26.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:39:26.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:39:26.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:39:26.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:39:27.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:39:27.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:39:27.480 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:39:27.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:39:27.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:39:27.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:39:27.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:39:28.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:39:28.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:39:28.716 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:39:28.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:39:28.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:39:28.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:39:28.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:39:29.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:39:29.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:39:29.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:39:29.959 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:39:29.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:39:29.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:39:30.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:39:31.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:39:31.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:39:31.193 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:39:31.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:39:31.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:39:31.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:39:31.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:39:32.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:39:32.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:39:32.431 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:39:32.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:39:32.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:39:32.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:39:32.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:39:33.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:39:33.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:39:33.663 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:39:33.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:39:33.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:39:33.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:39:34.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:39:35.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:39:35.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:39:35.200 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:39:35.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:39:35.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:39:35.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:39:35.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:39:36.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:39:36.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:39:36.436 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:39:36.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:39:36.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:39:36.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:39:37.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:39:38.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:39:38.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:39:38.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:39:38.100 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:39:38.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:39:38.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:39:38.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:39:39.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:39:39.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:39:39.334 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:39:39.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:39:39.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:39:39.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:39:39.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:39:40.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:39:40.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:39:40.564 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-06-03T05:39:40.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T05:39:40.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:39:40.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:39:40.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:39:41.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:39:41.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:39:41.802 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-06-03T05:39:41.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-03T05:39:41.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:39:41.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:39:42.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:39:43.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:39:43.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:39:43.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-03T05:39:43.043 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-06-03T05:39:43.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:39:43.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:39:43.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:39:44.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:39:44.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:39:44.275 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-06-03T05:39:44.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-03T05:39:44.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:39:44.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:39:44.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:39:45.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:39:45.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:39:45.510 INFO:tasks.workunit.client.0.smithi088.stdout:22 2024-06-03T05:39:45.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-03T05:39:45.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:39:45.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:39:45.810 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/22216749,v1:127.0.0.1:6809/22216749] [v2:127.0.0.1:6810/22216749,v1:127.0.0.1:6811/22216749] exists,up 2d18a088-38e7-4ff0-a35f-4edb3fb678c0 2024-06-03T05:39:45.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:39:45.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:39:45.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:39:45.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:80: TEST_check_version_health_1: run_osd td/ver-health 2 2024-06-03T05:39:45.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-06-03T05:39:45.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:39:45.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-03T05:39:45.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:39:45.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/2 2024-06-03T05:39:45.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6cd9f48e-d1df-426c-8c75-81e8c5105a2f --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-06-03T05:39:45.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:39:45.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:39:45.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:39:45.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/2' 2024-06-03T05:39:45.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2024-06-03T05:39:45.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:39:45.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:39:45.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-06-03T05:39:45.815 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:39:45.816 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:39:45.816 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:39:45.816 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:39:45.816 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:39:45.816 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184738 2024-06-03T05:39:45.816 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.184738/$cluster-$name.asok' 2024-06-03T05:39:45.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.184738/$cluster-$name.asok' 2024-06-03T05:39:45.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:39:45.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:39:45.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:39:45.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-06-03T05:39:45.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-06-03T05:39:45.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:39:45.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:39:45.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:39:45.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:39:45.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:39:45.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:39:45.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/2 2024-06-03T05:39:45.818 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:39:45.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=39b0c7e9-a456-431f-a92f-4964994f0697 2024-06-03T05:39:45.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 39b0c7e9-a456-431f-a92f-4964994f0697' 2024-06-03T05:39:45.818 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 39b0c7e9-a456-431f-a92f-4964994f0697 2024-06-03T05:39:45.819 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:39:45.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAhV11mPHuZMRAAngQjyarLJvA7ALGmcp61WA== 2024-06-03T05:39:45.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAhV11mPHuZMRAAngQjyarLJvA7ALGmcp61WA=="}' 2024-06-03T05:39:45.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 39b0c7e9-a456-431f-a92f-4964994f0697 -i td/ver-health/2/new.json 2024-06-03T05:39:47.038 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:39:47.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/2/new.json 2024-06-03T05:39:47.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=6cd9f48e-d1df-426c-8c75-81e8c5105a2f --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.184738/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAhV11mPHuZMRAAngQjyarLJvA7ALGmcp61WA== --osd-uuid 39b0c7e9-a456-431f-a92f-4964994f0697 2024-06-03T05:39:47.097 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:39:47.092+0000 7fafa3f397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:39:47.138 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:39:47.132+0000 7fafa3f397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:39:47.138 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:39:47.132+0000 7fafa3f397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:39:47.138 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:39:47.132+0000 7fafa3f397c0 -1 bluestore(td/ver-health/2/block) _read_bdev_label failed to open td/ver-health/2/block: (2) No such file or directory 2024-06-03T05:39:47.138 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:39:47.132+0000 7fafa3f397c0 -1 bluestore(td/ver-health/2/block) _read_bdev_label failed to open td/ver-health/2/block: (2) No such file or directory 2024-06-03T05:39:47.139 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:39:47.132+0000 7fafa3f397c0 -1 bluestore(td/ver-health/2/block) _read_bdev_label failed to open td/ver-health/2/block: (2) No such file or directory 2024-06-03T05:39:47.247 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:39:47.244+0000 7fafa3f397c0 -1 bluestore(td/ver-health/2) _read_fsid unparsable uuid 2024-06-03T05:39:53.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/2/keyring 2024-06-03T05:39:53.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:39:53.451 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-06-03T05:39:53.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-03T05:39:53.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:39:53.701 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-06-03T05:39:53.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-03T05:39:53.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=6cd9f48e-d1df-426c-8c75-81e8c5105a2f --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.184738/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:39:53.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:39:53.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:39:53.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:39:53.732 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:39:53.728+0000 7fe2d33267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:39:53.735 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:39:53.732+0000 7fe2d33267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:39:53.736 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:39:53.732+0000 7fe2d33267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:39:54.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-03T05:39:54.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:39:54.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T05:39:54.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:39:54.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:39:54.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:39:54.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:39:54.023 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:39:54.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:39:54.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:39:54.317 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:39:54.312+0000 7fe2d33267c0 -1 Falling back to public interface 2024-06-03T05:39:54.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:39:55.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:39:55.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:39:55.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:39:55.337 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:39:55.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:39:55.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:39:55.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:39:56.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:39:56.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:39:56.569 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:39:56.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:39:56.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:39:56.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:39:56.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:39:57.072 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:39:57.068+0000 7fe2d33267c0 -1 osd.2 0 log_to_monitors true 2024-06-03T05:39:57.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:39:57.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:39:57.804 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:39:57.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:39:57.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:39:57.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:39:58.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:39:59.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:39:59.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:39:59.037 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:39:59.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:39:59.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:39:59.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:39:59.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:40:00.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:40:00.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:40:00.271 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:40:00.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:40:00.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:40:00.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:40:00.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:40:01.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:40:01.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:40:01.586 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:40:01.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:40:01.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:40:01.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:40:01.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:40:02.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:40:02.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:40:02.818 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:40:02.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:40:02.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:40:02.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:40:03.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:40:04.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:40:04.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:40:04.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:40:04.053 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:40:04.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:40:04.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:40:04.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:40:05.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:40:05.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:40:05.283 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:40:05.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:40:05.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:40:05.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:40:05.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:40:06.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:40:06.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:40:06.520 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:40:06.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:40:06.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:40:06.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:40:06.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:40:07.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:40:07.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:40:07.753 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:40:07.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:40:07.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:40:07.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:40:07.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:40:08.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:40:08.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:40:08.985 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:40:08.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:40:08.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:40:08.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:40:09.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:40:10.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:40:10.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:40:10.222 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:40:10.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:40:10.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:40:10.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:40:10.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:40:11.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:40:11.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:40:11.455 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:40:11.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:40:11.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:40:11.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:40:11.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:40:12.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:40:12.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:40:12.697 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:40:12.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:40:12.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:40:12.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:40:12.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:40:13.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:40:13.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:40:13.934 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:40:13.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:40:13.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:40:13.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:40:14.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:40:15.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:40:15.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:40:15.170 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:40:15.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:40:15.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:40:15.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:40:15.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:40:16.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:40:16.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:40:16.402 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-06-03T05:40:16.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T05:40:16.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:40:16.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:40:16.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:40:17.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:40:17.655 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-06-03T05:40:17.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:40:17.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-03T05:40:17.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:40:17.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:40:17.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:40:18.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:40:18.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:40:18.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-03T05:40:18.892 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-06-03T05:40:18.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:40:18.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:40:19.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:40:20.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:40:20.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:40:20.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-03T05:40:20.253 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-06-03T05:40:20.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:40:20.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:40:20.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:40:21.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:40:21.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:40:21.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-03T05:40:21.487 INFO:tasks.workunit.client.0.smithi088.stdout:22 2024-06-03T05:40:21.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:40:21.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:40:21.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:40:22.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:40:22.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:40:22.722 INFO:tasks.workunit.client.0.smithi088.stdout:23 2024-06-03T05:40:22.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-03T05:40:22.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:40:22.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:40:23.570 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/54975629,v1:127.0.0.1:6817/54975629] [v2:127.0.0.1:6818/54975629,v1:127.0.0.1:6819/54975629] exists,up 39b0c7e9-a456-431f-a92f-4964994f0697 2024-06-03T05:40:23.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:40:23.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:40:23.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:40:23.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:82: TEST_check_version_health_1: sleep 5 2024-06-03T05:40:28.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:83: TEST_check_version_health_1: ceph health detail 2024-06-03T05:40:28.795 INFO:tasks.workunit.client.0.smithi088.stdout:HEALTH_ERR no active mgr 2024-06-03T05:40:28.795 INFO:tasks.workunit.client.0.smithi088.stdout:[ERR] MGR_DOWN: no active mgr 2024-06-03T05:40:28.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:85: TEST_check_version_health_1: ceph health detail 2024-06-03T05:40:28.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:85: TEST_check_version_health_1: grep DAEMON_OLD_VERSION 2024-06-03T05:40:29.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:87: TEST_check_version_health_1: kill_daemons td/ver-health KILL osd.1 2024-06-03T05:40:29.099 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:40:29.099 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:40:29.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:40:29.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:40:29.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:40:29.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:40:29.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:88: TEST_check_version_health_1: ceph_debug_version_for_testing=01.00.00-gversion-test 2024-06-03T05:40:29.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:88: TEST_check_version_health_1: activate_osd td/ver-health 1 2024-06-03T05:40:29.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-06-03T05:40:29.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T05:40:29.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-03T05:40:29.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T05:40:29.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/1 2024-06-03T05:40:29.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=6cd9f48e-d1df-426c-8c75-81e8c5105a2f --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-06-03T05:40:29.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:40:29.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:40:29.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:40:29.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/1' 2024-06-03T05:40:29.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/1/journal' 2024-06-03T05:40:29.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T05:40:29.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T05:40:29.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-06-03T05:40:29.210 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T05:40:29.210 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:40:29.210 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:40:29.211 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:40:29.211 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:40:29.211 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184738 2024-06-03T05:40:29.211 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.184738/$cluster-$name.asok' 2024-06-03T05:40:29.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.184738/$cluster-$name.asok' 2024-06-03T05:40:29.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:40:29.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-06-03T05:40:29.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-06-03T05:40:29.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:40:29.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:40:29.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:40:29.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:40:29.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T05:40:29.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T05:40:29.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/1 2024-06-03T05:40:29.213 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-06-03T05:40:29.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-03T05:40:29.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=6cd9f48e-d1df-426c-8c75-81e8c5105a2f --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.184738/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:40:29.214 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/1/whoami 2024-06-03T05:40:29.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-03T05:40:29.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T05:40:29.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T05:40:29.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T05:40:29.241 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:40:29.236+0000 7fa6fc3cd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:40:29.350 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:40:29.348+0000 7fa6fc3cd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:40:29.352 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:40:29.348+0000 7fa6fc3cd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:40:29.893 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:40:29.888+0000 7fa6fc3cd7c0 -1 Falling back to public interface 2024-06-03T05:40:29.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-03T05:40:29.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:40:29.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T05:40:29.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:40:29.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:40:29.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:40:29.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:40:29.964 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:40:29.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:40:29.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:40:30.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:40:31.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:40:31.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:40:31.383 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:40:31.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:40:31.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:40:31.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:40:31.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:40:32.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:40:32.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:40:32.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:40:32.616 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:40:32.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:40:32.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:40:32.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:40:33.299 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:40:33.296+0000 7fa6fc3cd7c0 -1 osd.1 15 log_to_monitors true 2024-06-03T05:40:33.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:40:33.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:40:33.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:40:33.859 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:40:33.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:40:33.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:40:34.144 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 18 up_thru 0 down_at 16 last_clean_interval [9,15) [v2:127.0.0.1:6808/1290725856,v1:127.0.0.1:6809/1290725856] [v2:127.0.0.1:6810/1290725856,v1:127.0.0.1:6811/1290725856] exists,up 2d18a088-38e7-4ff0-a35f-4edb3fb678c0 2024-06-03T05:40:34.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:40:34.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:40:34.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:40:34.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:90: TEST_check_version_health_1: wait_for_health_string 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-06-03T05:40:34.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-06-03T05:40:34.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2024-06-03T05:40:34.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2024-06-03T05:40:34.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2024-06-03T05:40:34.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2024-06-03T05:40:34.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-06-03T05:40:34.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-06-03T05:40:34.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-06-03T05:40:34.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-06-03T05:40:35.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-06-03T05:40:35.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-06-03T05:40:35.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-06-03T05:40:35.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-06-03T05:40:35.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-06-03T05:40:36.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-06-03T05:40:36.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-06-03T05:40:36.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-06-03T05:40:36.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-06-03T05:40:37.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-06-03T05:40:38.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-06-03T05:40:38.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-06-03T05:40:38.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-06-03T05:40:38.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-06-03T05:40:38.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2024-06-03T05:40:38.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2024-06-03T05:40:38.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2024-06-03T05:40:38.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2024-06-03T05:40:38.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2024-06-03T05:40:38.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:92: TEST_check_version_health_1: ceph health detail 2024-06-03T05:40:38.501 INFO:tasks.workunit.client.0.smithi088.stdout:HEALTH_WARN There is a daemon running an older version of ceph; no active mgr 2024-06-03T05:40:38.501 INFO:tasks.workunit.client.0.smithi088.stdout:[WRN] DAEMON_OLD_VERSION: There is a daemon running an older version of ceph 2024-06-03T05:40:38.501 INFO:tasks.workunit.client.0.smithi088.stdout: osd.1 is running an older version of ceph: 01.00.00-gversion-test 2024-06-03T05:40:38.501 INFO:tasks.workunit.client.0.smithi088.stdout:[WRN] MGR_DOWN: no active mgr 2024-06-03T05:40:38.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:94: TEST_check_version_health_1: ceph health detail 2024-06-03T05:40:38.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:94: TEST_check_version_health_1: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-06-03T05:40:38.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:95: TEST_check_version_health_1: ceph health detail 2024-06-03T05:40:38.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:95: TEST_check_version_health_1: grep -q '^[[]WRN[]] DAEMON_OLD_VERSION: There is a daemon running an older version of ceph' 2024-06-03T05:40:39.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:96: TEST_check_version_health_1: ceph health detail 2024-06-03T05:40:39.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:96: TEST_check_version_health_1: grep -q 'osd.1 is running an older version of ceph: 01.00.00-gversion-test' 2024-06-03T05:40:39.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:98: TEST_check_version_health_1: kill_daemons td/ver-health KILL osd.2 2024-06-03T05:40:39.428 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:40:39.428 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:40:39.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:40:39.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:40:39.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:40:39.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:40:39.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:99: TEST_check_version_health_1: ceph_debug_version_for_testing=01.00.00-gversion-test 2024-06-03T05:40:39.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:99: TEST_check_version_health_1: activate_osd td/ver-health 2 2024-06-03T05:40:39.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-06-03T05:40:39.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T05:40:39.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-06-03T05:40:39.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T05:40:39.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/2 2024-06-03T05:40:39.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=6cd9f48e-d1df-426c-8c75-81e8c5105a2f --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-06-03T05:40:39.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:40:39.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:40:39.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:40:39.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/2' 2024-06-03T05:40:39.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2024-06-03T05:40:39.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T05:40:39.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T05:40:39.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-06-03T05:40:39.541 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T05:40:39.541 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:40:39.541 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:40:39.541 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:40:39.541 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:40:39.541 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184738 2024-06-03T05:40:39.541 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.184738/$cluster-$name.asok' 2024-06-03T05:40:39.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.184738/$cluster-$name.asok' 2024-06-03T05:40:39.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:40:39.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-06-03T05:40:39.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-06-03T05:40:39.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:40:39.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:40:39.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:40:39.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:40:39.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T05:40:39.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T05:40:39.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/2 2024-06-03T05:40:39.542 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-06-03T05:40:39.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-06-03T05:40:39.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=6cd9f48e-d1df-426c-8c75-81e8c5105a2f --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.184738/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:40:39.546 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/2/whoami 2024-06-03T05:40:39.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-06-03T05:40:39.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T05:40:39.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T05:40:39.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T05:40:39.572 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:40:39.568+0000 7f23e60c17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:40:39.672 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:40:39.668+0000 7f23e60c17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:40:39.673 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:40:39.668+0000 7f23e60c17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:40:40.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-06-03T05:40:40.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:40:40.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T05:40:40.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:40:40.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:40:40.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:40:40.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:40:40.145 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:40:40.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:40:40.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:40:40.240 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:40:40.236+0000 7f23e60c17c0 -1 Falling back to public interface 2024-06-03T05:40:40.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:40:41.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:40:41.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:40:41.427 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:40:41.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:40:41.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:40:41.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:40:41.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:40:42.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:40:42.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:40:42.663 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:40:42.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:40:42.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:40:42.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:40:43.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:40:43.585 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:40:43.580+0000 7f23e60c17c0 -1 osd.2 17 log_to_monitors true 2024-06-03T05:40:44.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:40:44.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:40:44.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:40:44.015 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:40:44.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:40:44.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:40:44.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:40:45.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:40:45.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:40:45.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:40:45.332 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:40:45.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:40:45.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:40:45.564 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 21 up_thru 0 down_at 19 last_clean_interval [14,17) [v2:127.0.0.1:6816/1484985660,v1:127.0.0.1:6817/1484985660] [v2:127.0.0.1:6818/1484985660,v1:127.0.0.1:6819/1484985660] exists,up 39b0c7e9-a456-431f-a92f-4964994f0697 2024-06-03T05:40:45.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:40:45.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:40:45.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:40:45.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:100: TEST_check_version_health_1: kill_daemons td/ver-health KILL osd.0 2024-06-03T05:40:45.565 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:40:45.565 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:40:45.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:40:45.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:40:45.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:40:45.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:40:45.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:101: TEST_check_version_health_1: ceph_debug_version_for_testing=02.00.00-gversion-test 2024-06-03T05:40:45.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:101: TEST_check_version_health_1: activate_osd td/ver-health 0 2024-06-03T05:40:45.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-06-03T05:40:45.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-03T05:40:45.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-03T05:40:45.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-03T05:40:45.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/0 2024-06-03T05:40:45.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=6cd9f48e-d1df-426c-8c75-81e8c5105a2f --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-06-03T05:40:45.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:40:45.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:40:45.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:40:45.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/0' 2024-06-03T05:40:45.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2024-06-03T05:40:45.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-03T05:40:45.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-03T05:40:45.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-06-03T05:40:45.676 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-03T05:40:45.676 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:40:45.676 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:40:45.676 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:40:45.676 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:40:45.676 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184738 2024-06-03T05:40:45.678 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.184738/$cluster-$name.asok' 2024-06-03T05:40:45.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.184738/$cluster-$name.asok' 2024-06-03T05:40:45.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:40:45.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-06-03T05:40:45.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-06-03T05:40:45.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:40:45.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:40:45.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:40:45.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:40:45.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-03T05:40:45.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-03T05:40:45.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/0 2024-06-03T05:40:45.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-03T05:40:45.682 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:40:45.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=6cd9f48e-d1df-426c-8c75-81e8c5105a2f --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.184738/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:40:45.683 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/0/whoami 2024-06-03T05:40:45.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-03T05:40:45.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-03T05:40:45.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-03T05:40:45.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-03T05:40:45.706 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:40:45.704+0000 7fc410e247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:40:45.788 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:40:45.784+0000 7fc410e247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:40:45.789 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:40:45.784+0000 7fc410e247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:40:45.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-03T05:40:45.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:40:45.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:40:45.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:40:45.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:40:45.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:40:45.983 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:40:45.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:40:45.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:40:45.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:40:46.336 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:40:46.332+0000 7fc410e247c0 -1 Falling back to public interface 2024-06-03T05:40:46.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:40:47.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:40:47.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:40:47.467 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:40:47.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:40:47.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:40:47.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:40:47.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:40:48.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:40:48.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:40:48.973 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:40:48.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:40:48.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:40:48.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:40:49.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:40:49.439 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:40:49.436+0000 7fc410e247c0 -1 osd.0 21 log_to_monitors true 2024-06-03T05:40:50.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:40:50.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:40:50.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:40:50.207 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:40:50.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:40:50.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:40:50.544 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6800/1649999629,v1:127.0.0.1:6801/1649999629] [v2:127.0.0.1:6802/1649999629,v1:127.0.0.1:6803/1649999629] exists,up ae186b76-983b-4eb0-a39a-5b34a827afae 2024-06-03T05:40:50.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:40:50.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:40:50.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:40:50.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:103: TEST_check_version_health_1: wait_for_health_string 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-06-03T05:40:50.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-06-03T05:40:50.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2024-06-03T05:40:50.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2024-06-03T05:40:50.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2024-06-03T05:40:50.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2024-06-03T05:40:50.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-06-03T05:40:50.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-06-03T05:40:50.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-06-03T05:40:50.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-06-03T05:40:51.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-06-03T05:40:51.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-06-03T05:40:51.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-06-03T05:40:51.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-06-03T05:40:52.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-06-03T05:40:53.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-06-03T05:40:53.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-06-03T05:40:53.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-06-03T05:40:53.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-06-03T05:40:53.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2024-06-03T05:40:53.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2024-06-03T05:40:53.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2024-06-03T05:40:53.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2024-06-03T05:40:53.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2024-06-03T05:40:53.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:105: TEST_check_version_health_1: ceph health detail 2024-06-03T05:40:53.611 INFO:tasks.workunit.client.0.smithi088.stdout:HEALTH_ERR There are daemons running multiple old versions of ceph; no active mgr 2024-06-03T05:40:53.612 INFO:tasks.workunit.client.0.smithi088.stdout:[ERR] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph 2024-06-03T05:40:53.612 INFO:tasks.workunit.client.0.smithi088.stdout: osd.1 osd.2 are running an older version of ceph: 01.00.00-gversion-test 2024-06-03T05:40:53.612 INFO:tasks.workunit.client.0.smithi088.stdout: osd.0 is running an older version of ceph: 02.00.00-gversion-test 2024-06-03T05:40:53.612 INFO:tasks.workunit.client.0.smithi088.stdout:[WRN] MGR_DOWN: no active mgr 2024-06-03T05:40:53.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:106: TEST_check_version_health_1: ceph health detail 2024-06-03T05:40:53.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:106: TEST_check_version_health_1: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-06-03T05:40:53.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:107: TEST_check_version_health_1: ceph health detail 2024-06-03T05:40:53.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:107: TEST_check_version_health_1: grep -q '^[[]ERR[]] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph' 2024-06-03T05:40:54.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:108: TEST_check_version_health_1: grep -q 'osd.1 osd.2 are running an older version of ceph: 01.00.00-gversion-test' 2024-06-03T05:40:54.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:108: TEST_check_version_health_1: ceph health detail 2024-06-03T05:40:54.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:109: TEST_check_version_health_1: ceph health detail 2024-06-03T05:40:54.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:109: TEST_check_version_health_1: grep -q 'osd.0 is running an older version of ceph: 02.00.00-gversion-test' 2024-06-03T05:40:54.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:35: run: teardown td/ver-health 2024-06-03T05:40:54.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-06-03T05:40:54.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:40:54.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-06-03T05:40:54.871 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:40:54.871 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:40:54.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:40:54.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:40:54.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:40:54.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:40:54.980 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:40:54.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:40:54.981 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:40:54.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:40:54.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:40:54.983 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:40:54.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:40:54.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:40:54.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:40:54.984 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:40:54.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:40:54.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:40:54.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-06-03T05:40:55.002 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:40:55.002 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:40:55.002 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184738 2024-06-03T05:40:55.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.184738 2024-06-03T05:40:55.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:40:55.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:40:55.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:32: run: for func in $funcs 2024-06-03T05:40:55.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:33: run: setup td/ver-health 2024-06-03T05:40:55.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2024-06-03T05:40:55.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2024-06-03T05:40:55.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-06-03T05:40:55.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:40:55.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-06-03T05:40:55.004 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:40:55.004 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:40:55.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:40:55.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:40:55.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:40:55.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:40:55.006 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:40:55.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:40:55.006 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:40:55.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:40:55.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:40:55.008 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:40:55.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:40:55.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:40:55.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:40:55.009 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:40:55.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:40:55.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:40:55.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-06-03T05:40:55.011 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:40:55.011 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:40:55.011 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184738 2024-06-03T05:40:55.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.184738 2024-06-03T05:40:55.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:40:55.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:40:55.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2024-06-03T05:40:55.013 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:40:55.013 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:40:55.013 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184738 2024-06-03T05:40:55.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.184738 2024-06-03T05:40:55.016 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:40:55.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:40:55.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:40:55.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2024-06-03T05:40:55.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:34: run: TEST_check_version_health_2 td/ver-health 2024-06-03T05:40:55.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:114: TEST_check_version_health_2: local dir=td/ver-health 2024-06-03T05:40:55.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:117: TEST_check_version_health_2: CEPH_ARGS='--fsid=6cd9f48e-d1df-426c-8c75-81e8c5105a2f --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-06-03T05:40:55.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:119: TEST_check_version_health_2: setup td/ver-health 2024-06-03T05:40:55.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2024-06-03T05:40:55.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2024-06-03T05:40:55.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-06-03T05:40:55.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-03T05:40:55.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-06-03T05:40:55.017 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:40:55.017 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:40:55.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:40:55.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:40:55.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:40:55.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:40:55.017 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:40:55.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:40:55.018 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:40:55.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:40:55.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:40:55.019 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:40:55.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:40:55.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:40:55.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:40:55.021 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:40:55.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:40:55.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-03T05:40:55.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-06-03T05:40:55.023 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:40:55.023 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:40:55.023 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184738 2024-06-03T05:40:55.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.184738 2024-06-03T05:40:55.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:40:55.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:40:55.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2024-06-03T05:40:55.025 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-03T05:40:55.025 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:40:55.025 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184738 2024-06-03T05:40:55.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.184738 2024-06-03T05:40:55.027 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-03T05:40:55.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-03T05:40:55.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-03T05:40:55.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2024-06-03T05:40:55.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:122: TEST_check_version_health_2: run_mon td/ver-health a --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-06-03T05:40:55.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-06-03T05:40:55.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:40:55.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-03T05:40:55.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:40:55.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/a 2024-06-03T05:40:55.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ver-health/a --run-dir=td/ver-health --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-06-03T05:40:55.458 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:40:55.458 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:40:55.458 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:40:55.458 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:40:55.459 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:40:55.459 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184738 2024-06-03T05:40:55.459 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.184738/$cluster-$name.asok' 2024-06-03T05:40:55.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/a '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.184738/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-06-03T05:40:55.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:40:55.694 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-03T05:40:55.694 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:40:55.694 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:40:55.694 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:40:55.695 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:40:55.695 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:40:55.695 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:40:55.695 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:40:55.696 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:40:55.696 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:40:55.696 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184738 2024-06-03T05:40:55.696 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.184738/ceph-mon.a.asok 2024-06-03T05:40:55.696 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:40:55.696 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.184738/ceph-mon.a.asok config get fsid 2024-06-03T05:40:55.809 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-03T05:40:55.809 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:40:55.809 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-03T05:40:55.809 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:40:55.810 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-03T05:40:55.811 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-03T05:40:55.811 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-03T05:40:55.811 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:40:55.811 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:40:55.811 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:40:55.811 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184738 2024-06-03T05:40:55.811 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.184738/ceph-mon.a.asok 2024-06-03T05:40:55.811 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:40:55.811 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.184738/ceph-mon.a.asok config get mon_host 2024-06-03T05:40:55.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:123: TEST_check_version_health_2: run_mon td/ver-health b --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-06-03T05:40:55.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-06-03T05:40:55.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:40:55.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-06-03T05:40:55.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:40:55.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/b 2024-06-03T05:40:55.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ver-health/b --run-dir=td/ver-health --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-06-03T05:40:56.839 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:40:56.839 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:40:56.839 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:40:56.839 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:40:56.839 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:40:56.840 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184738 2024-06-03T05:40:56.840 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.184738/$cluster-$name.asok' 2024-06-03T05:40:56.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/b '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.184738/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-06-03T05:40:57.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:40:57.037 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-06-03T05:40:57.037 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:40:57.037 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-06-03T05:40:57.037 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:40:57.039 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:40:57.039 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-06-03T05:40:57.039 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-06-03T05:40:57.039 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-06-03T05:40:57.039 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:40:57.039 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:40:57.039 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184738 2024-06-03T05:40:57.039 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.184738/ceph-mon.b.asok 2024-06-03T05:40:57.040 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:40:57.040 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.184738/ceph-mon.b.asok config get fsid 2024-06-03T05:40:57.151 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-06-03T05:40:57.151 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:40:57.152 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-06-03T05:40:57.152 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:40:57.152 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:40:57.152 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-06-03T05:40:57.152 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-06-03T05:40:57.153 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-06-03T05:40:57.153 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:40:57.153 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:40:57.153 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184738 2024-06-03T05:40:57.153 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.184738/ceph-mon.b.asok 2024-06-03T05:40:57.154 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:40:57.154 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.184738/ceph-mon.b.asok config get mon_host 2024-06-03T05:40:57.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:124: TEST_check_version_health_2: run_osd td/ver-health 0 2024-06-03T05:40:57.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-06-03T05:40:57.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:40:57.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-03T05:40:57.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:40:57.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/0 2024-06-03T05:40:57.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6cd9f48e-d1df-426c-8c75-81e8c5105a2f --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-06-03T05:40:57.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:40:57.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:40:57.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:40:57.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/0' 2024-06-03T05:40:57.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2024-06-03T05:40:57.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:40:57.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:40:57.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-06-03T05:40:57.257 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:40:57.258 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:40:57.258 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:40:57.258 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:40:57.258 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:40:57.258 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184738 2024-06-03T05:40:57.258 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.184738/$cluster-$name.asok' 2024-06-03T05:40:57.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.184738/$cluster-$name.asok' 2024-06-03T05:40:57.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:40:57.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:40:57.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:40:57.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-06-03T05:40:57.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-06-03T05:40:57.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:40:57.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:40:57.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:40:57.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:40:57.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:40:57.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:40:57.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/0 2024-06-03T05:40:57.259 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:40:57.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e513c610-0376-4781-a175-acfce486609b 2024-06-03T05:40:57.260 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 e513c610-0376-4781-a175-acfce486609b 2024-06-03T05:40:57.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e513c610-0376-4781-a175-acfce486609b' 2024-06-03T05:40:57.260 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:40:57.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBpV11mvC+NEBAA3CeylgvMaODF9PDVV6Ozsg== 2024-06-03T05:40:57.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBpV11mvC+NEBAA3CeylgvMaODF9PDVV6Ozsg=="}' 2024-06-03T05:40:57.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e513c610-0376-4781-a175-acfce486609b -i td/ver-health/0/new.json 2024-06-03T05:41:03.648 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:41:03.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/0/new.json 2024-06-03T05:41:03.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=6cd9f48e-d1df-426c-8c75-81e8c5105a2f --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.184738/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBpV11mvC+NEBAA3CeylgvMaODF9PDVV6Ozsg== --osd-uuid e513c610-0376-4781-a175-acfce486609b 2024-06-03T05:41:03.701 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:41:03.696+0000 7f8fd43327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:41:03.703 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:41:03.700+0000 7f8fd43327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:41:03.704 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:41:03.700+0000 7f8fd43327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:41:03.704 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:41:03.700+0000 7f8fd43327c0 -1 bluestore(td/ver-health/0/block) _read_bdev_label failed to open td/ver-health/0/block: (2) No such file or directory 2024-06-03T05:41:03.705 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:41:03.700+0000 7f8fd43327c0 -1 bluestore(td/ver-health/0/block) _read_bdev_label failed to open td/ver-health/0/block: (2) No such file or directory 2024-06-03T05:41:03.705 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:41:03.700+0000 7f8fd43327c0 -1 bluestore(td/ver-health/0/block) _read_bdev_label failed to open td/ver-health/0/block: (2) No such file or directory 2024-06-03T05:41:03.825 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:41:03.820+0000 7f8fd43327c0 -1 bluestore(td/ver-health/0) _read_fsid unparsable uuid 2024-06-03T05:41:10.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/0/keyring 2024-06-03T05:41:10.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:41:10.262 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-06-03T05:41:10.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-03T05:41:10.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:41:10.507 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-06-03T05:41:10.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-03T05:41:10.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:41:10.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=6cd9f48e-d1df-426c-8c75-81e8c5105a2f --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.184738/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:41:10.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:41:10.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:41:10.540 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:41:10.536+0000 7f6bb3dd97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:41:10.542 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:41:10.540+0000 7f6bb3dd97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:41:10.543 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:41:10.540+0000 7f6bb3dd97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:41:10.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-03T05:41:10.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:41:10.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-03T05:41:10.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:41:10.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:41:10.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:41:10.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:41:10.794 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:41:10.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:41:10.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:41:11.084 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:41:11.080+0000 7f6bb3dd97c0 -1 Falling back to public interface 2024-06-03T05:41:11.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:41:12.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:41:12.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:41:12.099 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:41:12.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:41:12.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:41:12.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:41:12.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:41:13.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:41:13.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:41:13.336 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:41:13.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:41:13.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:41:13.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:41:13.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:41:13.992 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:41:13.988+0000 7f6bb3dd97c0 -1 osd.0 0 log_to_monitors true 2024-06-03T05:41:14.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:41:14.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:41:14.570 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:41:14.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:41:14.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:41:14.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:41:14.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:41:15.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:41:15.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:41:15.988 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:41:15.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:41:15.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:41:15.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:41:16.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:41:17.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:41:17.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:41:17.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:41:17.223 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:41:17.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:41:17.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:41:17.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:41:18.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:41:18.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:41:18.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:41:18.455 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:41:18.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:41:18.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:41:18.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:41:19.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:41:19.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:41:19.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:41:19.689 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:41:19.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:41:19.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:41:19.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:41:20.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:41:20.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:41:20.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:41:20.921 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:41:20.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:41:20.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:41:21.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:41:22.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:41:22.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:41:22.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:41:22.154 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:41:22.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:41:22.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:41:22.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:41:23.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:41:23.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:41:23.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:41:23.386 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:41:23.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:41:23.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:41:23.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:41:24.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:41:24.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:41:24.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:41:24.623 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:41:24.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:41:24.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:41:24.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:41:25.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:41:25.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:41:25.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:41:25.859 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:41:25.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:41:25.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:41:26.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:41:27.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:41:27.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:41:27.096 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:41:27.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:41:27.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:41:27.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:41:27.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:41:28.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:41:28.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:41:28.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:41:28.327 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:41:28.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:41:28.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:41:28.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:41:29.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:41:29.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:41:29.558 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:41:29.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:41:29.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:41:29.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:41:29.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:41:30.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:41:30.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:41:30.791 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:41:30.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:41:30.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:41:30.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:41:31.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:41:32.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:41:32.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:41:32.026 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:41:32.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:41:32.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:41:32.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:41:32.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:41:33.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:41:33.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:41:33.262 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-06-03T05:41:33.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T05:41:33.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:41:33.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:41:33.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:41:34.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:41:34.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:41:34.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-03T05:41:34.496 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-06-03T05:41:34.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:41:34.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:41:34.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:41:35.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:41:35.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:41:35.737 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-06-03T05:41:35.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-03T05:41:35.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:41:35.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:41:35.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:41:36.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:41:36.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:41:36.970 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-06-03T05:41:36.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-03T05:41:36.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:41:36.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:41:37.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:41:38.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:41:38.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:41:38.205 INFO:tasks.workunit.client.0.smithi088.stdout:22 2024-06-03T05:41:38.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-03T05:41:38.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:41:38.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:41:38.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:41:39.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:41:39.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:41:39.441 INFO:tasks.workunit.client.0.smithi088.stdout:23 2024-06-03T05:41:39.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-03T05:41:39.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:41:39.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-03T05:41:39.706 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2518985309,v1:127.0.0.1:6801/2518985309] [v2:127.0.0.1:6802/2518985309,v1:127.0.0.1:6803/2518985309] exists,up e513c610-0376-4781-a175-acfce486609b 2024-06-03T05:41:39.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:41:39.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:41:39.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:41:39.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:125: TEST_check_version_health_2: run_osd td/ver-health 1 2024-06-03T05:41:39.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-06-03T05:41:39.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:41:39.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-03T05:41:39.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:41:39.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/1 2024-06-03T05:41:39.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6cd9f48e-d1df-426c-8c75-81e8c5105a2f --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-06-03T05:41:39.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:41:39.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:41:39.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:41:39.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/1' 2024-06-03T05:41:39.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/1/journal' 2024-06-03T05:41:39.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:41:39.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:41:39.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-06-03T05:41:39.710 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:41:39.710 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:41:39.710 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:41:39.713 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:41:39.713 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:41:39.713 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184738 2024-06-03T05:41:39.713 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.184738/$cluster-$name.asok' 2024-06-03T05:41:39.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.184738/$cluster-$name.asok' 2024-06-03T05:41:39.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:41:39.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:41:39.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:41:39.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-06-03T05:41:39.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-06-03T05:41:39.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:41:39.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:41:39.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:41:39.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:41:39.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:41:39.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:41:39.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/1 2024-06-03T05:41:39.715 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:41:39.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ca86c0c9-d5cf-4793-a195-a98a1037d114 2024-06-03T05:41:39.715 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 ca86c0c9-d5cf-4793-a195-a98a1037d114 2024-06-03T05:41:39.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 ca86c0c9-d5cf-4793-a195-a98a1037d114' 2024-06-03T05:41:39.716 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:41:39.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCTV11mFwNxKxAAhuL1XbEob3vGsnWvSCJ1OQ== 2024-06-03T05:41:39.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCTV11mFwNxKxAAhuL1XbEob3vGsnWvSCJ1OQ=="}' 2024-06-03T05:41:39.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ca86c0c9-d5cf-4793-a195-a98a1037d114 -i td/ver-health/1/new.json 2024-06-03T05:41:40.155 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:41:40.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/1/new.json 2024-06-03T05:41:40.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=6cd9f48e-d1df-426c-8c75-81e8c5105a2f --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.184738/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCTV11mFwNxKxAAhuL1XbEob3vGsnWvSCJ1OQ== --osd-uuid ca86c0c9-d5cf-4793-a195-a98a1037d114 2024-06-03T05:41:40.208 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:41:40.204+0000 7f594bc6e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:41:40.210 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:41:40.208+0000 7f594bc6e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:41:40.211 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:41:40.208+0000 7f594bc6e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:41:40.212 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:41:40.208+0000 7f594bc6e7c0 -1 bluestore(td/ver-health/1/block) _read_bdev_label failed to open td/ver-health/1/block: (2) No such file or directory 2024-06-03T05:41:40.212 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:41:40.208+0000 7f594bc6e7c0 -1 bluestore(td/ver-health/1/block) _read_bdev_label failed to open td/ver-health/1/block: (2) No such file or directory 2024-06-03T05:41:40.212 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:41:40.208+0000 7f594bc6e7c0 -1 bluestore(td/ver-health/1/block) _read_bdev_label failed to open td/ver-health/1/block: (2) No such file or directory 2024-06-03T05:41:40.310 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:41:40.304+0000 7f594bc6e7c0 -1 bluestore(td/ver-health/1) _read_fsid unparsable uuid 2024-06-03T05:41:46.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/1/keyring 2024-06-03T05:41:46.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:41:46.746 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-06-03T05:41:46.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-03T05:41:46.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:41:46.995 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-06-03T05:41:46.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-03T05:41:46.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=6cd9f48e-d1df-426c-8c75-81e8c5105a2f --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.184738/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:41:46.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:41:46.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:41:46.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:41:47.026 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:41:47.024+0000 7f8decc527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:41:47.029 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:41:47.024+0000 7f8decc527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:41:47.030 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:41:47.024+0000 7f8decc527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:41:47.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-03T05:41:47.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:41:47.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-03T05:41:47.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:41:47.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:41:47.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:41:47.297 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:41:47.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:41:47.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:41:47.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:41:47.568 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:41:47.564+0000 7f8decc527c0 -1 Falling back to public interface 2024-06-03T05:41:47.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:41:48.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:41:48.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:41:48.630 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:41:48.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:41:48.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:41:48.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:41:48.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:41:49.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:41:49.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:41:49.858 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:41:49.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:41:49.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:41:49.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:41:50.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:41:50.351 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:41:50.348+0000 7f8decc527c0 -1 osd.1 0 log_to_monitors true 2024-06-03T05:41:51.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:41:51.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:41:51.088 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:41:51.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:41:51.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:41:51.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:41:51.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:41:52.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:41:52.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:41:52.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:41:52.398 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:41:52.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:41:52.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:41:52.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:41:53.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:41:53.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:41:53.634 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:41:53.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:41:53.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:41:53.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:41:53.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:41:54.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:41:54.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:41:54.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:41:54.866 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:41:54.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:41:54.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:41:55.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:41:56.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:41:56.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:41:56.100 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:41:56.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:41:56.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:41:56.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:41:56.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:41:57.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:41:57.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:41:57.339 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:41:57.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:41:57.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:41:57.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:41:57.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:41:58.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:41:58.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:41:58.573 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:41:58.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:41:58.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:41:58.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:41:58.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:41:59.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:41:59.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:41:59.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:41:59.807 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:41:59.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:41:59.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:42:00.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:42:01.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:42:01.038 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:42:01.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:42:01.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:42:01.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:42:01.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:42:01.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:42:02.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:42:02.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:42:02.266 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:42:02.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:42:02.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:42:02.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:42:02.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:42:03.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:42:03.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:42:03.582 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:42:03.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:42:03.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:42:03.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:42:03.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:42:04.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:42:04.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:42:04.867 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:42:04.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:42:04.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:42:04.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:42:05.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:42:06.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:42:06.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:42:06.492 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:42:06.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:42:06.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:42:06.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:42:06.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:42:07.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:42:07.726 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:42:07.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:42:07.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:42:07.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:42:07.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:42:07.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:42:08.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:42:08.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:42:08.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:42:08.957 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:42:08.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:42:08.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:42:09.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:42:10.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:42:10.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:42:10.190 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-06-03T05:42:10.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T05:42:10.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:42:10.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:42:10.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:42:11.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:42:11.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:42:11.423 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-06-03T05:42:11.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-03T05:42:11.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:42:11.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:42:11.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:42:12.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:42:12.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:42:12.659 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-06-03T05:42:12.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-03T05:42:12.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:42:12.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:42:12.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:42:13.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:42:13.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:42:13.893 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-06-03T05:42:13.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-03T05:42:13.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:42:13.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:42:14.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:42:15.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:42:15.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:42:15.132 INFO:tasks.workunit.client.0.smithi088.stdout:22 2024-06-03T05:42:15.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-03T05:42:15.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:42:15.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-03T05:42:15.550 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/2618489688,v1:127.0.0.1:6809/2618489688] [v2:127.0.0.1:6810/2618489688,v1:127.0.0.1:6811/2618489688] exists,up ca86c0c9-d5cf-4793-a195-a98a1037d114 2024-06-03T05:42:15.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:42:15.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:42:15.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:42:15.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:126: TEST_check_version_health_2: run_osd td/ver-health 2 2024-06-03T05:42:15.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-06-03T05:42:15.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-03T05:42:15.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-03T05:42:15.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-03T05:42:15.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/2 2024-06-03T05:42:15.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6cd9f48e-d1df-426c-8c75-81e8c5105a2f --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-06-03T05:42:15.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-03T05:42:15.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-03T05:42:15.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-03T05:42:15.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/2' 2024-06-03T05:42:15.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2024-06-03T05:42:15.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-03T05:42:15.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-03T05:42:15.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-06-03T05:42:15.553 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-03T05:42:15.553 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:42:15.553 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:42:15.553 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:42:15.553 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:42:15.554 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184738 2024-06-03T05:42:15.555 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.184738/$cluster-$name.asok' 2024-06-03T05:42:15.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.184738/$cluster-$name.asok' 2024-06-03T05:42:15.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-03T05:42:15.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-03T05:42:15.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-03T05:42:15.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-06-03T05:42:15.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-06-03T05:42:15.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-03T05:42:15.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-03T05:42:15.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-03T05:42:15.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-03T05:42:15.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-03T05:42:15.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-03T05:42:15.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/2 2024-06-03T05:42:15.556 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-03T05:42:15.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ce6307f3-a21e-4072-a6a2-560ad3afc38a 2024-06-03T05:42:15.557 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 ce6307f3-a21e-4072-a6a2-560ad3afc38a 2024-06-03T05:42:15.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 ce6307f3-a21e-4072-a6a2-560ad3afc38a' 2024-06-03T05:42:15.557 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-03T05:42:15.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC3V11mDEc+IhAA/+BAYwqjDWrr9atZPo3/KA== 2024-06-03T05:42:15.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC3V11mDEc+IhAA/+BAYwqjDWrr9atZPo3/KA=="}' 2024-06-03T05:42:15.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ce6307f3-a21e-4072-a6a2-560ad3afc38a -i td/ver-health/2/new.json 2024-06-03T05:42:16.338 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:42:16.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/2/new.json 2024-06-03T05:42:16.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=6cd9f48e-d1df-426c-8c75-81e8c5105a2f --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.184738/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC3V11mDEc+IhAA/+BAYwqjDWrr9atZPo3/KA== --osd-uuid ce6307f3-a21e-4072-a6a2-560ad3afc38a 2024-06-03T05:42:16.397 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:42:16.392+0000 7f20067627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:42:16.483 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:42:16.480+0000 7f20067627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:42:16.486 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:42:16.480+0000 7f20067627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:42:16.486 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:42:16.480+0000 7f20067627c0 -1 bluestore(td/ver-health/2/block) _read_bdev_label failed to open td/ver-health/2/block: (2) No such file or directory 2024-06-03T05:42:16.486 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:42:16.480+0000 7f20067627c0 -1 bluestore(td/ver-health/2/block) _read_bdev_label failed to open td/ver-health/2/block: (2) No such file or directory 2024-06-03T05:42:16.486 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:42:16.480+0000 7f20067627c0 -1 bluestore(td/ver-health/2/block) _read_bdev_label failed to open td/ver-health/2/block: (2) No such file or directory 2024-06-03T05:42:16.574 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:42:16.568+0000 7f20067627c0 -1 bluestore(td/ver-health/2) _read_fsid unparsable uuid 2024-06-03T05:42:23.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/2/keyring 2024-06-03T05:42:23.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-03T05:42:23.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-03T05:42:23.305 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-06-03T05:42:23.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-03T05:42:23.548 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-06-03T05:42:23.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-03T05:42:23.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=6cd9f48e-d1df-426c-8c75-81e8c5105a2f --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.184738/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-03T05:42:23.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-03T05:42:23.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-03T05:42:23.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-03T05:42:23.580 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:42:23.576+0000 7f3c49c587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:42:23.582 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:42:23.576+0000 7f3c49c587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:42:23.584 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:42:23.580+0000 7f3c49c587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-03T05:42:23.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-03T05:42:23.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-03T05:42:23.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-03T05:42:23.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-03T05:42:23.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-03T05:42:23.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:42:23.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-03T05:42:23.880 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-06-03T05:42:23.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:42:23.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:42:24.172 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:42:24.168+0000 7f3c49c587c0 -1 Falling back to public interface 2024-06-03T05:42:24.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:42:25.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:42:25.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:42:25.212 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-06-03T05:42:25.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-03T05:42:25.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:42:25.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:42:25.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:42:26.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:42:26.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:42:26.447 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-06-03T05:42:26.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-03T05:42:26.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:42:26.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:42:26.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:42:26.911 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:42:26.908+0000 7f3c49c587c0 -1 osd.2 0 log_to_monitors true 2024-06-03T05:42:27.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:42:27.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:42:27.684 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-06-03T05:42:27.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-03T05:42:27.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:42:27.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:42:28.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:42:29.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:42:29.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:42:29.069 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-06-03T05:42:29.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-03T05:42:29.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:42:29.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:42:29.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:42:30.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:42:30.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:42:30.303 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-06-03T05:42:30.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-03T05:42:30.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:42:30.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:42:30.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:42:31.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:42:31.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:42:31.536 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-06-03T05:42:31.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-03T05:42:31.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:42:31.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:42:31.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:42:32.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:42:32.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:42:32.761 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-06-03T05:42:32.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-03T05:42:32.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:42:32.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:42:32.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:42:33.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:42:33.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:42:33.995 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-06-03T05:42:33.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-03T05:42:33.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:42:33.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:42:34.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:42:35.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:42:35.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:42:35.232 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-06-03T05:42:35.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-03T05:42:35.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:42:35.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:42:35.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:42:36.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:42:36.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:42:36.458 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-06-03T05:42:36.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-03T05:42:36.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:42:36.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:42:36.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:42:37.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:42:37.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:42:37.695 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-06-03T05:42:37.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-03T05:42:37.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:42:37.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:42:37.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:42:38.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:42:38.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:42:38.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-03T05:42:38.920 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-06-03T05:42:38.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:42:38.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:42:39.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:42:40.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:42:40.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:42:40.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-03T05:42:40.155 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-06-03T05:42:40.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:42:40.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:42:40.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:42:41.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:42:41.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:42:41.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-03T05:42:41.389 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-06-03T05:42:41.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:42:41.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:42:41.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:42:42.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:42:42.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:42:42.622 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-06-03T05:42:42.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-03T05:42:42.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:42:42.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:42:42.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:42:43.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:42:43.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:42:43.859 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-06-03T05:42:43.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-03T05:42:43.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:42:43.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:42:44.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:42:45.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:42:45.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:42:45.096 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-06-03T05:42:45.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-03T05:42:45.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:42:45.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:42:45.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:42:46.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:42:46.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:42:46.324 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-06-03T05:42:46.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-03T05:42:46.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:42:46.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:42:46.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:42:47.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:42:47.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:42:47.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-03T05:42:47.552 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-06-03T05:42:47.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:42:47.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:42:47.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:42:48.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:42:48.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:42:48.785 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-06-03T05:42:48.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-03T05:42:48.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:42:48.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:42:49.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:42:50.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:42:50.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:42:50.020 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-06-03T05:42:50.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-03T05:42:50.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:42:50.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:42:50.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:42:51.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:42:51.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:42:51.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-03T05:42:51.256 INFO:tasks.workunit.client.0.smithi088.stdout:22 2024-06-03T05:42:51.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:42:51.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:42:51.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-03T05:42:52.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-03T05:42:52.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-03T05:42:52.591 INFO:tasks.workunit.client.0.smithi088.stdout:23 2024-06-03T05:42:52.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-03T05:42:52.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-03T05:42:52.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-03T05:42:52.825 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/650136673,v1:127.0.0.1:6817/650136673] [v2:127.0.0.1:6818/650136673,v1:127.0.0.1:6819/650136673] exists,up ce6307f3-a21e-4072-a6a2-560ad3afc38a 2024-06-03T05:42:52.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-03T05:42:52.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-03T05:42:52.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-03T05:42:52.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:127: TEST_check_version_health_2: run_mgr td/ver-health x 2024-06-03T05:42:52.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ver-health 2024-06-03T05:42:52.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T05:42:52.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-03T05:42:52.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T05:42:52.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ver-health/x 2024-06-03T05:42:52.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T05:42:53.231 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T05:42:53.231 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:42:53.231 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:42:53.232 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:42:53.232 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:42:53.232 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184738 2024-06-03T05:42:53.232 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.184738/$cluster-$name.asok' 2024-06-03T05:42:53.233 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:42:53.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ver-health/x '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.184738/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:42:53.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:128: TEST_check_version_health_2: run_mgr td/ver-health y 2024-06-03T05:42:53.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ver-health 2024-06-03T05:42:53.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-03T05:42:53.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=y 2024-06-03T05:42:53.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-03T05:42:53.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ver-health/y 2024-06-03T05:42:53.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-03T05:42:53.576 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-03T05:42:53.576 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:42:53.576 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:42:53.577 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:42:53.577 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:42:53.577 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184738 2024-06-03T05:42:53.577 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.184738/$cluster-$name.asok' 2024-06-03T05:42:53.577 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:42:53.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id y --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ver-health/y '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.184738/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-03T05:42:53.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:129: TEST_check_version_health_2: run_mds td/ver-health m 2024-06-03T05:42:53.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ver-health 2024-06-03T05:42:53.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-06-03T05:42:53.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=m 2024-06-03T05:42:53.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-06-03T05:42:53.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ver-health/m 2024-06-03T05:42:53.605 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-06-03T05:42:53.605 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:42:53.606 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:42:53.606 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:42:53.606 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:42:53.606 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184738 2024-06-03T05:42:53.606 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.184738/$cluster-$name.asok' 2024-06-03T05:42:53.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id m --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ver-health/m '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.184738/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' 2024-06-03T05:42:53.631 INFO:tasks.workunit.client.0.smithi088.stdout:starting mds.m at 2024-06-03T05:42:53.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:130: TEST_check_version_health_2: run_mds td/ver-health n 2024-06-03T05:42:53.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ver-health 2024-06-03T05:42:53.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-06-03T05:42:53.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=n 2024-06-03T05:42:53.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-06-03T05:42:53.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ver-health/n 2024-06-03T05:42:53.635 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-06-03T05:42:53.635 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:42:53.635 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:42:53.636 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:42:53.636 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:42:53.636 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184738 2024-06-03T05:42:53.636 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.184738/$cluster-$name.asok' 2024-06-03T05:42:53.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id n --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ver-health/n '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.184738/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' 2024-06-03T05:42:53.693 INFO:tasks.workunit.client.0.smithi088.stdout:starting mds.n at 2024-06-03T05:42:53.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:132: TEST_check_version_health_2: sleep 5 2024-06-03T05:42:58.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:133: TEST_check_version_health_2: ceph health detail 2024-06-03T05:42:59.369 INFO:tasks.workunit.client.0.smithi088.stdout:HEALTH_OK 2024-06-03T05:42:59.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:135: TEST_check_version_health_2: ceph health detail 2024-06-03T05:42:59.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:135: TEST_check_version_health_2: grep DAEMON_OLD_VERSION 2024-06-03T05:42:59.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:137: TEST_check_version_health_2: kill_daemons td/ver-health KILL mon.b 2024-06-03T05:42:59.938 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:42:59.938 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:42:59.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:42:59.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:42:59.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:43:00.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:43:00.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:138: TEST_check_version_health_2: ceph_debug_version_for_testing=01.00.00-gversion-test 2024-06-03T05:43:00.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:138: TEST_check_version_health_2: run_mon td/ver-health b --mon_warn_older_version_delay=0 2024-06-03T05:43:00.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-06-03T05:43:00.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-03T05:43:00.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-06-03T05:43:00.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-03T05:43:00.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/b 2024-06-03T05:43:00.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ver-health/b --run-dir=td/ver-health --mon_warn_older_version_delay=0 2024-06-03T05:43:00.071 INFO:tasks.workunit.client.0.smithi088.stderr:2024-06-03T05:43:00.068+0000 7f1b9b77ec80 -1 'td/ver-health/b' already exists and is not empty: monitor may already exist 2024-06-03T05:43:00.073 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-03T05:43:00.073 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:43:00.073 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:43:00.074 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:43:00.074 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:43:00.074 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184738 2024-06-03T05:43:00.074 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.184738/$cluster-$name.asok' 2024-06-03T05:43:00.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/b '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.184738/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --mon_warn_older_version_delay=0 2024-06-03T05:43:00.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-03T05:43:00.321 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-06-03T05:43:00.321 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:43:00.321 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-06-03T05:43:00.321 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-03T05:43:00.322 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-03T05:43:00.323 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-06-03T05:43:00.323 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-06-03T05:43:00.323 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-06-03T05:43:00.323 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:43:00.323 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:43:00.323 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184738 2024-06-03T05:43:00.323 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.184738/ceph-mon.b.asok 2024-06-03T05:43:00.323 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:43:00.323 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.184738/ceph-mon.b.asok config get fsid 2024-06-03T05:43:00.426 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-06-03T05:43:00.426 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-03T05:43:00.426 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-06-03T05:43:00.426 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-03T05:43:00.428 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-03T05:43:00.428 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-06-03T05:43:00.428 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-06-03T05:43:00.428 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-06-03T05:43:00.428 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-03T05:43:00.428 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:43:00.428 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184738 2024-06-03T05:43:00.428 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.184738/ceph-mon.b.asok 2024-06-03T05:43:00.428 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-03T05:43:00.428 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.184738/ceph-mon.b.asok config get mon_host 2024-06-03T05:43:00.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:142: TEST_check_version_health_2: kill_daemons td/ver-health KILL mds.m 2024-06-03T05:43:00.540 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:43:00.540 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:43:00.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:43:00.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:43:00.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:43:00.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:43:00.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:143: TEST_check_version_health_2: ceph_debug_version_for_testing=01.00.00-gversion-test 2024-06-03T05:43:00.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:143: TEST_check_version_health_2: run_mds td/ver-health m 2024-06-03T05:43:00.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ver-health 2024-06-03T05:43:00.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-06-03T05:43:00.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=m 2024-06-03T05:43:00.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-06-03T05:43:00.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ver-health/m 2024-06-03T05:43:00.649 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-06-03T05:43:00.649 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-03T05:43:00.649 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-03T05:43:00.650 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-03T05:43:00.650 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:43:00.650 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184738 2024-06-03T05:43:00.650 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.184738/$cluster-$name.asok' 2024-06-03T05:43:00.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id m --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ver-health/m '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.184738/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' 2024-06-03T05:43:00.887 INFO:tasks.workunit.client.0.smithi088.stdout:starting mds.m at 2024-06-03T05:43:00.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:145: TEST_check_version_health_2: wait_for_health_string 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-06-03T05:43:00.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_WARN .*There are daemons running an older version of ceph' 2024-06-03T05:43:00.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2024-06-03T05:43:00.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2024-06-03T05:43:00.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2024-06-03T05:43:00.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2024-06-03T05:43:00.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-06-03T05:43:00.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-06-03T05:43:00.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-06-03T05:43:01.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-06-03T05:43:02.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-06-03T05:43:02.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-06-03T05:43:02.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-06-03T05:43:02.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-06-03T05:43:03.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2024-06-03T05:43:03.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2024-06-03T05:43:03.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2024-06-03T05:43:03.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2024-06-03T05:43:03.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2024-06-03T05:43:03.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:147: TEST_check_version_health_2: ceph health detail 2024-06-03T05:43:03.656 INFO:tasks.workunit.client.0.smithi088.stdout:HEALTH_WARN There are daemons running an older version of ceph 2024-06-03T05:43:03.657 INFO:tasks.workunit.client.0.smithi088.stdout:[WRN] DAEMON_OLD_VERSION: There are daemons running an older version of ceph 2024-06-03T05:43:03.657 INFO:tasks.workunit.client.0.smithi088.stdout: mon.b mds. are running an older version of ceph: 01.00.00-gversion-test 2024-06-03T05:43:03.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:149: TEST_check_version_health_2: ceph health detail 2024-06-03T05:43:03.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:149: TEST_check_version_health_2: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-06-03T05:43:04.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:150: TEST_check_version_health_2: ceph health detail 2024-06-03T05:43:04.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:150: TEST_check_version_health_2: grep -q '^[[]WRN[]] DAEMON_OLD_VERSION: There are daemons running an older version of ceph' 2024-06-03T05:43:04.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:151: TEST_check_version_health_2: ceph health detail 2024-06-03T05:43:04.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:151: TEST_check_version_health_2: grep -q 'mon.b mds.m are running an older version of ceph: 01.00.00-gversion-test' 2024-06-03T05:43:05.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:151: TEST_check_version_health_2: return 1 2024-06-03T05:43:05.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:34: run: return 1 2024-06-03T05:43:05.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: code=1 2024-06-03T05:43:05.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/ver-health 1 2024-06-03T05:43:05.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-06-03T05:43:05.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=1 2024-06-03T05:43:05.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-06-03T05:43:05.522 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-03T05:43:05.522 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-03T05:43:05.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-03T05:43:05.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-03T05:43:05.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-03T05:43:05.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-03T05:43:05.835 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-03T05:43:05.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-03T05:43:05.837 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-03T05:43:05.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-03T05:43:05.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-03T05:43:05.838 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-03T05:43:05.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:43:05.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-03T05:43:05.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-03T05:43:05.840 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:43:05.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:43:05.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 1 = 1 ']' 2024-06-03T05:43:05.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: '[' -n '' ']' 2024-06-03T05:43:05.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:194: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2024-06-03T05:43:05.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:195: teardown: mv td/ver-health/mds.m.log td/ver-health/mds.n.log td/ver-health/mgr.x.log td/ver-health/mgr.y.log td/ver-health/mon.a.log td/ver-health/mon.b.log td/ver-health/osd.0.log td/ver-health/osd.1.log td/ver-health/osd.2.log /home/ubuntu/cephtest/archive/log 2024-06-03T05:43:05.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-06-03T05:43:05.858 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-03T05:43:05.858 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-03T05:43:05.858 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184738 2024-06-03T05:43:05.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.184738 2024-06-03T05:43:05.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-03T05:43:05.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-03T05:43:05.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 1 2024-06-03T05:43:05.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-03T05:43:05.861 INFO:tasks.workunit:Stopping ['misc'] on client.0... 2024-06-03T05:43:05.861 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-06-03T05:43:06.197 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_dfcc4b3c2c55532c4c04fa47551c9df1dffdf746/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_dfcc4b3c2c55532c4c04fa47551c9df1dffdf746/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_dfcc4b3c2c55532c4c04fa47551c9df1dffdf746/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test misc/ver-health.sh) on smithi088 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dfcc4b3c2c55532c4c04fa47551c9df1dffdf746 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh' 2024-06-03T05:43:06.380 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=ded48abcebbc4e4d9c8c73eee6b71065 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_dfcc4b3c2c55532c4c04fa47551c9df1dffdf746/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_dfcc4b3c2c55532c4c04fa47551c9df1dffdf746/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_dfcc4b3c2c55532c4c04fa47551c9df1dffdf746/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test misc/ver-health.sh) on smithi088 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dfcc4b3c2c55532c4c04fa47551c9df1dffdf746 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh' 2024-06-03T05:43:06.382 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-06-03T05:43:06.390 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-06-03T05:43:06.391 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-03T05:43:06.430 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-06-03T05:43:06.483 INFO:teuthology.task.clock:Checking final clock skew... 2024-06-03T05:43:06.483 DEBUG:teuthology.orchestra.run.smithi088:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-03T05:43:06.502 INFO:teuthology.orchestra.run.smithi088.stdout: remote refid st t when poll reach delay offset jitter 2024-06-03T05:43:06.502 INFO:teuthology.orchestra.run.smithi088.stdout:============================================================================== 2024-06-03T05:43:06.502 INFO:teuthology.orchestra.run.smithi088.stdout:+hv01.front.sepi 67.205.162.81 3 u 4 128 377 0.128 +1.724 0.524 2024-06-03T05:43:06.502 INFO:teuthology.orchestra.run.smithi088.stdout:*hv02.front.sepi 63.231.80.2 3 u 2 128 377 0.173 +3.356 0.370 2024-06-03T05:43:06.502 INFO:teuthology.orchestra.run.smithi088.stdout:+hv03.front.sepi 74.6.168.72 3 u 16 128 377 0.173 +1.957 0.556 2024-06-03T05:43:06.502 INFO:teuthology.orchestra.run.smithi088.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-06-03T05:43:06.503 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-06-03T05:43:06.547 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-06-03T05:43:06.547 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-06-03T05:43:06.606 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-06-03T05:43:06.698 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-06-03T05:43:06.781 INFO:teuthology.task.internal:Duration was 4278.437505 seconds 2024-06-03T05:43:06.781 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-06-03T05:43:06.856 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-06-03T05:43:06.856 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-03T05:43:06.902 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-06-03T05:43:06.902 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi088.front.sepia.ceph.com 2024-06-03T05:43:06.903 DEBUG:teuthology.orchestra.run.smithi088:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-03T05:43:06.955 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-06-03T05:43:06.955 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-03T05:43:07.033 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-06-03T05:43:07.034 DEBUG:teuthology.orchestra.run.smithi088:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-03T05:43:07.127 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-06-03T05:43:07.137 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-06-03T05:43:07.137 DEBUG:teuthology.orchestra.run.smithi088:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-03T05:43:07.149 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-06-03T05:43:07.171 DEBUG:teuthology.orchestra.run.smithi088:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:43:07.201 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = core 2024-06-03T05:43:07.219 DEBUG:teuthology.orchestra.run.smithi088:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:43:07.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-03T05:43:07.270 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-06-03T05:43:07.280 INFO:teuthology.task.internal:Transferring archived files... 2024-06-03T05:43:07.281 DEBUG:teuthology.misc:Transferring archived files from smithi088:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-06-02_21:00:03-rados-squid-distro-default-smithi/7738364/remote/smithi088 2024-06-03T05:43:07.282 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-03T05:43:08.061 INFO:teuthology.task.internal:Removing archive directory... 2024-06-03T05:43:08.062 DEBUG:teuthology.orchestra.run.smithi088:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-03T05:43:08.106 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-06-03T05:43:08.430 INFO:teuthology.task.internal:Not uploading archives. 2024-06-03T05:43:08.430 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-06-03T05:43:08.458 INFO:teuthology.task.internal:Tidying up after the test... 2024-06-03T05:43:08.458 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-03T05:43:08.465 INFO:teuthology.orchestra.run.smithi088.stdout: 658641 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Jun 3 05:43 /home/ubuntu/cephtest 2024-06-03T05:43:08.466 INFO:teuthology.orchestra.run.smithi088.stdout: 671751 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Jun 3 04:41 /home/ubuntu/cephtest/mnt.0 2024-06-03T05:43:08.466 INFO:teuthology.orchestra.run.smithi088.stdout: 673777 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Jun 3 05:38 /home/ubuntu/cephtest/mnt.0/client.0 2024-06-03T05:43:08.466 INFO:teuthology.orchestra.run.smithi088.stdout: 686700 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Jun 3 05:38 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-06-03T05:43:08.466 INFO:teuthology.orchestra.run.smithi088.stdout: 686701 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Jun 3 05:43 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2024-06-03T05:43:08.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-03T05:43:08.467 INFO:teuthology.orchestra.run.smithi088.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-06-03T05:43:08.467 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-06-03T05:43:08.472 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-06-03T05:43:08.512 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-06-03T05:43:08.591 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi088 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-06-03T05:43:08.595 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/misc} duration: 4278.437504529953 failure_reason: 'Command failed (workunit test misc/ver-health.sh) on smithi088 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dfcc4b3c2c55532c4c04fa47551c9df1dffdf746 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh''' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=ded48abcebbc4e4d9c8c73eee6b71065 status: fail success: false 2024-06-03T05:43:08.595 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-03T05:43:08.673 INFO:teuthology.run:FAIL